# CoFRIDA: Self-Supervised Fine-Tuning for Human-Robot Co-Painting

arXiv: https://arxiv.org/abs/2402.13442 · HTML: https://mlsanity.com/paper/2402.13442/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 Feb 2024.

## Built on by (most cited first, 8 of 8)

- [SketchAgent: Language-Driven Sequential Sketch Generation](https://mlsanity.com/paper/2411.17673/) (2024)
- [RT-Cache: Training-Free Retrieval for Real-Time Manipulation](https://mlsanity.com/paper/2505.09040/) (2025)
- [RoboSVG: A Unified Framework for Interactive SVG Generation with Multi-modal Guidance](https://mlsanity.com/paper/2510.22684/) (2025)
- [AnimatePainter: A Self-Supervised Rendering Framework for Reconstructing Painting Process](https://mlsanity.com/paper/2503.17029/) (2025)
- [Birth of a Painting: Differentiable Brushstroke Reconstruction](https://mlsanity.com/paper/2511.13191/) (2025)
- [Patch-Based Spatial Authorship Attribution in Human-Robot Collaborative Paintings](https://mlsanity.com/paper/2602.17030/) (2026)
- [Data-Efficient Brushstroke Generation with Diffusion Models for Oil Painting](https://mlsanity.com/paper/2603.01103/) (2026)
- [IMPASTO: Integrating Model-Based Planning with Learned Dynamics Models for Robotic Oil Painting Reproduction](https://mlsanity.com/paper/2603.29315/) (2026)

## Abstract (arXiv)

Prior robot painting and drawing work, such as FRIDA, has focused on decreasing the sim-to-real gap and expanding input modalities for users, but the interaction with these systems generally exists only in the input stages. To support interactive, human-robot collaborative painting, we introduce the Collaborative FRIDA (CoFRIDA) robot painting framework, which can co-paint by modifying and engaging with content already painted by a human collaborator. To improve text-image alignment, FRIDA's major weakness, our system uses pre-trained text-to-image models; however, pre-trained models in the context of real-world co-painting do not perform well because they (1) do not understand the constraints and abilities of the robot and (2) cannot perform co-painting without making unrealistic edits to the canvas and overwriting content. We propose a self-supervised fine-tuning procedure that can tackle both issues, allowing the use of pre-trained state-of-the-art text-image alignment models with robots to enable co-painting in the physical world. Our open-source approach, CoFRIDA, creates paintings and drawings that match the input text prompt more clearly than FRIDA, both from a blank canvas and one with human created work. More generally, our fine-tuning procedure successfully encodes the robot's constraints and abilities into a foundation model, showcasing promising results as an effective method for reducing sim-to-real gaps.