AI-powered robot, FRIDA, helps humans to make art

FRIDA, an AI-powered robotic arm fixed to a paintbrush, is collaborating with artists to make art. The AI puts a brush on the canvas when prompted to paint a picture. Like OpenAI’s ChatGPT and DALL-E 2, FRIDA employs AI models to create results. The AI simulates painting an image with brush strokes and evaluates its progress with machine learning. 

Named after Frida Kahlo, FRIDA is an abbreviation for Framework and Robotics Initiative for Developing Arts. Peter Schaldenbrand, a School of Computer Science Ph.D. student at the Robotics Institute (RI) of Carnegie Mellon University (CMU), and RI faculty members Jim McCann and Jean Oh are the project coordinators.

“FRIDA is a robotic painting system, but FRIDA is not an artist. FRIDA is not generating the ideas to communicate. FRIDA is a system that an artist could collaborate with. The artist can specify high-level goals for FRIDA, and then FRIDA can execute them.”

Peter Schaldenbrand, student at Robotics Institute, CMU

How FRIDA works

FRIDA uses AI and machine learning numerous times throughout its artistic process. It spends the first hour feeling acquainted with its paintbrush. Then, it leverages sophisticated vision-language models, like OpenAI’s Contrastive Language-Image Pre-Training (CLIP), trained on enormous datasets linking images and texts taken from the web, to comprehend the input. AI systems use these models to produce new text or graphics in response to input.

Large vision-language models are used by image-generating systems, such as OpenAI‘s DALL-E 2, to create digital images. With its robotic system, FRIDA goes beyond Open AI’s DALL-E-2’s large vision-language models to create physical paintings. Reducing the simulation-to-real gap is arguably the most significant technical challenge in creating a physical image. The simulation-to-real gap marks the difference between FRIDA’s outlook in simulation and what ends on the canvas. 

The robot’s brush strokes train the simulator to portray the mirror and emulate the physical characteristics of the robot as well as the painting materials.

FRIDA is a multicultural AI

FRIDA’s creators also want to tackle limitations facing existing large vision-language models. The researchers injected the models with headlines from the latest news stories to get them updated with happenings worldwide. They also trained them with texts and images from diverse cultures to avoid Western bias.

First-year master’s students Beverley-Claire Okogwu and Zhixuan Liu and Visiting Master’s students Youngsik Yun and Youeun Shin led the multicultural effort. They trained data contributions from Japan, China, Korea, Norway, Mexico, Vietnam, Nigeria, and other countries.

More FRIDA features

After a user specifies a high-level notion of the painting they wish to create on FRIDA, the AI utilizes machine learning to simulate and plan a painting to meet their expectations. The AI exhibits a color pallet on the monitor for the user to mix and feed the robot. Jiaying Wei, a Master’s student in architecture, and Eunsu Kang, Machine Learning Department faculty, are developing automatic paint mixing.

The robot often photographs the painting with an overhead camera. In a process that takes hours, The graphic helps FRIDA analyze its progress and adjust its plan.

“People wonder if FRIDA will take artists’ jobs, but the main goal of the FRIDA project is quite the opposite. We want to really promote human creativity through FRIDA,” Oh said. “For instance, I wanted to be an artist. Now, I can collaborate with FRIDA to express my ideas in painting.”

You can get more information on FRIDA on its website. The team will present its newest project research, “FRIDA: A Collaborative Robot Painter With a Differentiable, Real2Sim2Real Planning Environment,” in May 2023 at the IEEE International Conference on Robotics and Automation in London in May. 

The featured image was taken from Carnegie Mellon University