# PaintCopilot: Modeling Painting as Autonomous Artistic Continuation

arXiv: https://arxiv.org/abs/2605.20941 · HTML: https://mlsanity.com/paper/2605.20941/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 26 of its 59 references that are in the map.
- First posted on arXiv 20 May 2026.

## Builds on (most cited first, 26 of 26)

- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [InstructPix2Pix: Learning to Follow Image Editing Instructions](https://mlsanity.com/paper/2211.09800/) (2022)
- [Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Prediction](https://mlsanity.com/paper/2404.02905/) (2024)
- [BiSeNet: Bilateral Segmentation Network for Real-time Semantic Segmentation](https://mlsanity.com/paper/1808.00897/) (2018)
- [Deep ViT Features as Dense Visual Descriptors](https://mlsanity.com/paper/2112.05814/) (2021)
- [A Neural Representation of Sketch Drawings](https://mlsanity.com/paper/1704.03477/) (2017)
- [Rethinking Inductive Biases for Surface Normal Estimation](https://mlsanity.com/paper/2403.00712/) (2024)
- [CSGNet: Neural Shape Parser for Constructive Solid Geometry](https://mlsanity.com/paper/1712.08290/) (2017)
- [Synthesizing Programs for Images using Reinforced Adversarial Learning](https://mlsanity.com/paper/1804.01118/) (2018)
- [Learning to Paint With Model-based Deep Reinforcement Learning](https://mlsanity.com/paper/1903.04411/) (2019)
- [Paint Transformer: Feed Forward Neural Painting with Stroke Prediction](https://mlsanity.com/paper/2108.03798/) (2021)
- [Stylized Neural Painting](https://mlsanity.com/paper/2011.08114/) (2020)
- [Rethinking Style Transfer: From Pixels to Parameterized Brushstrokes](https://mlsanity.com/paper/2103.17185/) (2021)
- [ProcessPainter: Learn Painting Process from Sequence Data](https://mlsanity.com/paper/2406.06062/) (2024)
- [Stroke-based Neural Painting and Stylization with Dynamically Predicted Painting Region](https://mlsanity.com/paper/2309.03504/) (2023)
- [Approximate Bayesian Image Interpretation using Generative Probabilistic Graphics Programs](https://mlsanity.com/paper/1307.0060/) (2013)
- [Where Do I 'Add the Egg'?: Exploring Agency and Ownership in AI Creative Co-Writing Systems](https://mlsanity.com/paper/2509.15440/) (2025)
- [Inverse Painting: Reconstructing The Painting Process](https://mlsanity.com/paper/2409.20556/) (2024)
- [Stroke-based Rendering: From Heuristics to Deep Learning](https://mlsanity.com/paper/2302.00595/) (2022)
- [How Do Human Creators Embrace Human-AI Co-Creation? A Perspective on Human Agency of Screenwriters](https://mlsanity.com/paper/2602.06327/) (2026)
- [HistoryPalette: Supporting Exploration and Reuse of Past Alternatives in Image Generation and Editing](https://mlsanity.com/paper/2501.04163/) (2025)
- [Vectorized Region Based Brush Strokes for Artistic Rendering](https://mlsanity.com/paper/2506.09969/) (2025)
- [Hyperstroke: A Novel High-quality Stroke Representation for Assistive Artistic Drawing](https://mlsanity.com/paper/2408.09348/) (2024)
- [AnimatePainter: A Self-Supervised Rendering Framework for Reconstructing Painting Process](https://mlsanity.com/paper/2503.17029/) (2025)
- [InkIdeator: Supporting Chinese-Style Visual Design Ideation via AI-Infused Exploration of Chinese Paintings](https://mlsanity.com/paper/2601.18193/) (2026)

## Abstract (arXiv)

Existing neural painting methods are target-driven: given a reference image, strokes are optimized to reconstruct it, fixing the outcome before painting begins. We instead ask whether a model can predict plausible painting actions when the final image is unknown, a problem we call autonomous artistic continuation. PaintCopilot realizes this through three lightweight, task-specific models: a Target Predictor that infers a provisional visual target directly from the evolving canvas, a Stroke Predictor that formulates brushstroke generation as flow matching to capture multiple plausible continuations, and a Region Sampler that generates stroke sequences within artist-specified regions via a conditional VAE, trained on a constructed dataset of 3000 portraits with stroke-level supervision. This lets AI autonomously act on an artwork whose direction is still open, through interruptible, revisable delegation mechanisms spanning individual strokes, extended sequences, bounded regions, and prior stroke history. A two-week deployment with ten artists suggests that creative agency was shaped not simply by how much work artists delegated, but by their ability to continually negotiate the direction, scope, and persistence of AI participation.