# Learning to Paint With Model-based Deep Reinforcement Learning

arXiv: https://arxiv.org/abs/1903.04411 · HTML: https://mlsanity.com/paper/1903.04411/

## Key facts

- Cited by 71 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 Mar 2019.

## Built on by (most cited first, 30 of 71)

- [Im2Vec: Synthesizing Vector Graphics without Vector Supervision](https://mlsanity.com/paper/2102.02798/) (2021)
- [Learning to generate line drawings that convey geometry and semantics](https://mlsanity.com/paper/2203.12691/) (2022)
- [Paint Transformer: Feed Forward Neural Painting with Stroke Prediction](https://mlsanity.com/paper/2108.03798/) (2021)
- [Procedure Planning in Instructional Videos via Contextual Modeling and Model-based Policy Learning](https://mlsanity.com/paper/2110.01770/) (2021)
- [Towards Artistic Image Aesthetics Assessment: a Large-scale Dataset and a New Method](https://mlsanity.com/paper/2303.15166/) (2023)
- [Rethinking Style Transfer: From Pixels to Parameterized Brushstrokes](https://mlsanity.com/paper/2103.17185/) (2021)
- [Modern Evolution Strategies for Creativity: Fitting Concrete Images and Abstract Concepts](https://mlsanity.com/paper/2109.08857/) (2021)
- [Torch.fx: Practical Program Capture and Transformation for Deep Learning in Python](https://mlsanity.com/paper/2112.08429/) (2021)
- [ProcessPainter: Learn Painting Process from Sequence Data](https://mlsanity.com/paper/2406.06062/) (2024)
- [High-Fidelity Guided Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2211.17084/) (2022)
- [CLIPVG: Text-Guided Image Manipulation Using Differentiable Vector Graphics](https://mlsanity.com/paper/2212.02122/) (2022)
- [StyleCLIPDraw: Coupling Content and Style in Text-to-Drawing Translation](https://mlsanity.com/paper/2202.12362/) (2022)
- [Position, Padding and Predictions: A Deeper Look at Position Information in CNNs](https://mlsanity.com/paper/2101.12322/) (2021)
- [Creativity and Machine Learning: A Survey](https://mlsanity.com/paper/2104.02726/) (2021)
- [Brick-by-Brick: Combinatorial Construction with Deep Reinforcement Learning](https://mlsanity.com/paper/2110.15481/) (2021)
- [Emergent Graphical Conventions in a Visual Communication Game](https://mlsanity.com/paper/2111.14210/) (2021)
- [DeepVecFont: Synthesizing High-quality Vector Fonts via Dual-modality Learning](https://mlsanity.com/paper/2110.06688/) (2021)
- [Stroke-based Neural Painting and Stylization with Dynamically Predicted Painting Region](https://mlsanity.com/paper/2309.03504/) (2023)
- [Reinforcement Learning for Generative AI: State of the Art, Opportunities and Open Research Challenges](https://mlsanity.com/paper/2308.00031/) (2023)
- [Toward Modeling Creative Processes for Algorithmic Painting](https://mlsanity.com/paper/2205.01605/) (2022)
- [Im2Oil: Stroke-Based Oil Painting Rendering with Linearly Controllable Fineness Via Adaptive Sampling](https://mlsanity.com/paper/2209.13219/) (2022)
- [PICASSO: A Feed-Forward Framework for Parametric Inference of CAD Sketches via Rendering Self-Supervision](https://mlsanity.com/paper/2407.13394/) (2024)
- [ModelLight: Model-Based Meta-Reinforcement Learning for Traffic Signal Control](https://mlsanity.com/paper/2111.08067/) (2021)
- [Collaborative Neural Rendering using Anime Character Sheets](https://mlsanity.com/paper/2207.05378/) (2022)
- [Explore 3D Dance Generation via Reward Model from Automatically-Ranked Demonstrations](https://mlsanity.com/paper/2312.11442/) (2023)
- [Intelli-Paint: Towards Developing Human-like Painting Agents](https://mlsanity.com/paper/2112.08930/) (2021)
- [Paint2Pix: Interactive Painting based Progressive Image Synthesis and Editing](https://mlsanity.com/paper/2208.08092/) (2022)
- [Inverse Painting: Reconstructing The Painting Process](https://mlsanity.com/paper/2409.20556/) (2024)
- [PaintNet: Unstructured Multi-Path Learning from 3D Point Clouds for Robotic Spray Painting](https://mlsanity.com/paper/2211.06930/) (2022)
- [Learning Geometry-aware Representations by Sketching](https://mlsanity.com/paper/2304.08204/) (2023)

## Abstract (arXiv)

We show how to teach machines to paint like human painters, who can use a small number of strokes to create fantastic paintings. By employing a neural renderer in model-based Deep Reinforcement Learning (DRL), our agents learn to determine the position and color of each stroke and make long-term plans to decompose texture-rich images into strokes. Experiments demonstrate that excellent visual effects can be achieved using hundreds of strokes. The training process does not require the experience of human painters or stroke tracking data. The code is available at https://github.com/hzwer/ICCV2019-LearningToPaint.