# Inverse Painting: Reconstructing The Painting Process

arXiv: https://arxiv.org/abs/2409.20556 · HTML: https://mlsanity.com/paper/2409.20556/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 47 references that are in the map.
- First posted on arXiv 30 Sep 2024.

## Builds on (most cited first, 24 of 24)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection](https://mlsanity.com/paper/2303.05499/) (2023)
- [Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets](https://mlsanity.com/paper/2311.15127/) (2023)
- [Animate Anyone: Consistent and Controllable Image-to-Video Synthesis for Character Animation](https://mlsanity.com/paper/2311.17117/) (2023)
- [Paint by Example: Exemplar-based Image Editing with Diffusion Models](https://mlsanity.com/paper/2211.13227/) (2022)
- [A Neural Representation of Sketch Drawings](https://mlsanity.com/paper/1704.03477/) (2017)
- [Synthesizing Programs for Images using Reinforced Adversarial Learning](https://mlsanity.com/paper/1804.01118/) (2018)
- [Learning to Paint With Model-based Deep Reinforcement Learning](https://mlsanity.com/paper/1903.04411/) (2019)
- [pix2gestalt: Amodal Segmentation by Synthesizing Wholes](https://mlsanity.com/paper/2401.14398/) (2024)
- [Paint Transformer: Feed Forward Neural Painting with Stroke Prediction](https://mlsanity.com/paper/2108.03798/) (2021)
- [On the Content Bias in Fr'echet Video Distance](https://mlsanity.com/paper/2404.12391/) (2024)
- [DALLE-2 is Seeing Double: Flaws in Word-to-Concept Mapping in Text2Image Models](https://mlsanity.com/paper/2210.10606/) (2022)
- [Stylized Neural Painting](https://mlsanity.com/paper/2011.08114/) (2020)
- [Rethinking Style Transfer: From Pixels to Parameterized Brushstrokes](https://mlsanity.com/paper/2103.17185/) (2021)
- [Stroke-based Neural Painting and Stylization with Dynamically Predicted Painting Region](https://mlsanity.com/paper/2309.03504/) (2023)
- [Combining Semantic Guidance and Deep Reinforcement Learning For Generating Human Level Paintings](https://mlsanity.com/paper/2011.12589/) (2020)
- [Painting Many Pasts: Synthesizing Time Lapse Videos of Paintings](https://mlsanity.com/paper/2001.01026/) (2020)
- [Intelli-Paint: Towards Developing Human-like Painting Agents](https://mlsanity.com/paper/2112.08930/) (2021)
- [Unsupervised Image to Sequence Translation with Canvas-Drawer Networks](https://mlsanity.com/paper/1809.08340/) (2018)
- [Total Selfie: Generating Full-Body Selfies](https://mlsanity.com/paper/2308.14740/) (2023)
- [PaintBot: A Reinforcement Learning Approach for Natural Media Painting](https://mlsanity.com/paper/1904.02201/) (2019)

## Built on by (most cited first, 6 of 6)

- [Loom: Diffusion-Transformer for Interleaved Generation](https://mlsanity.com/paper/2512.18254/) (2025)
- [AnimatePainter: A Self-Supervised Rendering Framework for Reconstructing Painting Process](https://mlsanity.com/paper/2503.17029/) (2025)
- [PPJudge: Towards Human-Aligned Assessment of Artistic Painting Process](https://mlsanity.com/paper/2507.09242/) (2025)
- [Birth of a Painting: Differentiable Brushstroke Reconstruction](https://mlsanity.com/paper/2511.13191/) (2025)
- [Loomis Painter: Reconstructing the Painting Process](https://mlsanity.com/paper/2511.17344/) (2025)
- [PaintCopilot: Modeling Painting as Autonomous Artistic Continuation](https://mlsanity.com/paper/2605.20941/) (2026)

## Abstract (arXiv)

Given an input painting, we reconstruct a time-lapse video of how it may have been painted. We formulate this as an autoregressive image generation problem, in which an initially blank "canvas" is iteratively updated. The model learns from real artists by training on many painting videos. Our approach incorporates text and region understanding to define a set of painting "instructions" and updates the canvas with a novel diffusion-based renderer. The method extrapolates beyond the limited, acrylic style paintings on which it has been trained, showing plausible results for a wide range of artistic styles and genres.