# Stroke-based Neural Painting and Stylization with Dynamically Predicted Painting Region

arXiv: https://arxiv.org/abs/2309.03504 · HTML: https://mlsanity.com/paper/2309.03504/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 31 references that are in the map.
- First posted on arXiv 7 Sep 2023.

## Builds on (most cited first, 16 of 16)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [Progressive Growing of GANs for Improved Quality, Stability, and Variation](https://mlsanity.com/paper/1710.10196/) (2017)
- [Improved Training of Wasserstein GANs](https://mlsanity.com/paper/1704.00028/) (2017)
- [Arbitrary Style Transfer in Real-time with Adaptive Instance Normalization](https://mlsanity.com/paper/1703.06868/) (2017)
- [Generating Sequences With Recurrent Neural Networks](https://mlsanity.com/paper/1308.0850/) (2013)
- [A Neural Representation of Sketch Drawings](https://mlsanity.com/paper/1704.03477/) (2017)
- [AdaAttN: Revisit Attention Mechanism in Arbitrary Neural Style Transfer](https://mlsanity.com/paper/2108.03647/) (2021)
- [Synthesizing Programs for Images using Reinforced Adversarial Learning](https://mlsanity.com/paper/1804.01118/) (2018)
- [Learning to Paint With Model-based Deep Reinforcement Learning](https://mlsanity.com/paper/1903.04411/) (2019)
- [Paint Transformer: Feed Forward Neural Painting with Stroke Prediction](https://mlsanity.com/paper/2108.03798/) (2021)
- [Stylized Neural Painting](https://mlsanity.com/paper/2011.08114/) (2020)
- [Rethinking Style Transfer: From Pixels to Parameterized Brushstrokes](https://mlsanity.com/paper/2103.17185/) (2021)
- [Combining Semantic Guidance and Deep Reinforcement Learning For Generating Human Level Paintings](https://mlsanity.com/paper/2011.12589/) (2020)
- [Im2Oil: Stroke-Based Oil Painting Rendering with Linearly Controllable Fineness Via Adaptive Sampling](https://mlsanity.com/paper/2209.13219/) (2022)

## Built on by (most cited first, 14 of 14)

- [LayerTracer: Cognitive-Aligned Layered SVG Synthesis via Diffusion Transformer](https://mlsanity.com/paper/2502.01105/) (2025)
- [ProcessPainter: Learn Painting Process from Sequence Data](https://mlsanity.com/paper/2406.06062/) (2024)
- [UniCombine: Unified Multi-Conditional Combination with Diffusion Transformer](https://mlsanity.com/paper/2503.09277/) (2025)
- [SAMVG: A Multi-stage Image Vectorization Model with the Segment-Anything Model](https://mlsanity.com/paper/2311.05276/) (2023)
- [Inverse Painting: Reconstructing The Painting Process](https://mlsanity.com/paper/2409.20556/) (2024)
- [Stroke of Surprise: Progressive Semantic Illusions in Vector Sketching](https://mlsanity.com/paper/2602.12280/) (2026)
- [Vectorized Region Based Brush Strokes for Artistic Rendering](https://mlsanity.com/paper/2506.09969/) (2025)
- [Text2VDM: Text to Vector Displacement Maps for Expressive and Interactive 3D Sculpting](https://mlsanity.com/paper/2502.20045/) (2025)
- [Look, Compare and Draw: Differential Query Transformer for Automatic Oil Painting](https://mlsanity.com/paper/2603.27720/) (2026)
- [MambaPainter: Neural Stroke-Based Rendering in a Single Step](https://mlsanity.com/paper/2410.12524/) (2024)
- [Stroke-based Cyclic Amplifier: Image Super-Resolution at Arbitrary Ultra-Large Scales](https://mlsanity.com/paper/2506.10774/) (2025)
- [PaintFlow: A Unified Framework for Interactive Oil Paintings Editing and Generation](https://mlsanity.com/paper/2512.08534/) (2025)
- [Data-Efficient Brushstroke Generation with Diffusion Models for Oil Painting](https://mlsanity.com/paper/2603.01103/) (2026)
- [PaintCopilot: Modeling Painting as Autonomous Artistic Continuation](https://mlsanity.com/paper/2605.20941/) (2026)

## Abstract (arXiv)

Stroke-based rendering aims to recreate an image with a set of strokes. Most existing methods render complex images using an uniform-block-dividing strategy, which leads to boundary inconsistency artifacts. To solve the problem, we propose Compositional Neural Painter, a novel stroke-based rendering framework which dynamically predicts the next painting region based on the current canvas, instead of dividing the image plane uniformly into painting regions. We start from an empty canvas and divide the painting process into several steps. At each step, a compositor network trained with a phasic RL strategy first predicts the next painting region, then a painter network trained with a WGAN discriminator predicts stroke parameters, and a stroke renderer paints the strokes onto the painting region of the current canvas. Moreover, we extend our method to stroke-based style transfer with a novel differentiable distance transform loss, which helps preserve the structure of the input image during stroke-based stylization. Extensive experiments show our model outperforms the existing models in both stroke-based neural painting and stroke-based stylization. Code is available at https://github.com/sjtuplayer/Compositional_Neural_Painter