# Look, Compare and Draw: Differential Query Transformer for Automatic Oil Painting

arXiv: https://arxiv.org/abs/2603.27720 · HTML: https://mlsanity.com/paper/2603.27720/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 63 references that are in the map.
- First posted on arXiv 29 Mar 2026.

## Builds on (most cited first, 29 of 29)

- [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018)
- [Progressive Growing of GANs for Improved Quality, Stability, and Variation](https://mlsanity.com/paper/1710.10196/) (2017)
- [Perceptual Losses for Real-Time Style Transfer and Super-Resolution](https://mlsanity.com/paper/1603.08155/) (2016)
- [Improved Training of Wasserstein GANs](https://mlsanity.com/paper/1704.00028/) (2017)
- [Generating Sequences With Recurrent Neural Networks](https://mlsanity.com/paper/1308.0850/) (2013)
- [An Intriguing Failing of Convolutional Neural Networks and the CoordConv Solution](https://mlsanity.com/paper/1807.03247/) (2018)
- [Style Aligned Image Generation via Shared Attention](https://mlsanity.com/paper/2312.02133/) (2023)
- [Joint Discriminative and Generative Learning for Person Re-identification](https://mlsanity.com/paper/1904.07223/) (2019)
- [CLIPDraw: Exploring Text-to-Drawing Synthesis through Language-Image Encoders](https://mlsanity.com/paper/2106.14843/) (2021)
- [Implicit Style-Content Separation using B-LoRA](https://mlsanity.com/paper/2403.14572/) (2024)
- [Synthesizing Programs for Images using Reinforced Adversarial Learning](https://mlsanity.com/paper/1804.01118/) (2018)
- [Learning to Paint With Model-based Deep Reinforcement Learning](https://mlsanity.com/paper/1903.04411/) (2019)
- [Paint Transformer: Feed Forward Neural Painting with Stroke Prediction](https://mlsanity.com/paper/2108.03798/) (2021)
- [Stylized Neural Painting](https://mlsanity.com/paper/2011.08114/) (2020)
- [Rethinking Style Transfer: From Pixels to Parameterized Brushstrokes](https://mlsanity.com/paper/2103.17185/) (2021)
- [ProcessPainter: Learn Painting Process from Sequence Data](https://mlsanity.com/paper/2406.06062/) (2024)
- [TextIR: A Simple Framework for Text-based Editable Image Restoration](https://mlsanity.com/paper/2302.14736/) (2023)
- [Jointly Harnessing Prior Structures and Temporal Consistency for Sign Language Video Generation](https://mlsanity.com/paper/2207.03714/) (2022)
- [Stroke-based Neural Painting and Stylization with Dynamically Predicted Painting Region](https://mlsanity.com/paper/2309.03504/) (2023)
- [Combining Semantic Guidance and Deep Reinforcement Learning For Generating Human Level Paintings](https://mlsanity.com/paper/2011.12589/) (2020)
- [Human-Guided Image Generation for Expanding Small-Scale Training Image Datasets](https://mlsanity.com/paper/2412.16839/) (2024)
- [Im2Oil: Stroke-Based Oil Painting Rendering with Linearly Controllable Fineness Via Adaptive Sampling](https://mlsanity.com/paper/2209.13219/) (2022)
- [Progressive Text-to-3D Generation for Automatic 3D Prototyping](https://mlsanity.com/paper/2309.14600/) (2023)
- [NPRportrait 1.0: A Three-Level Benchmark for Non-Photorealistic Rendering of Portraits](https://mlsanity.com/paper/2009.00633/) (2020)
- [CreativeSynth: Cross-Art-Attention for Artistic Image Synthesis with Multimodal Diffusion](https://mlsanity.com/paper/2401.14066/) (2024)
- [Unsupervised Image to Sequence Translation with Canvas-Drawer Networks](https://mlsanity.com/paper/1809.08340/) (2018)
- [3D Magic Mirror: Clothing Reconstruction from a Single Image via a Causal Perspective](https://mlsanity.com/paper/2204.13096/) (2022)
- [Drawing out of Distribution with Neuro-Symbolic Generative Models](https://mlsanity.com/paper/2206.01829/) (2022)
- [Segmentation-Based Parametric Painting](https://mlsanity.com/paper/2311.14271/) (2023)

## Built on by (most cited first, 1 of 1)

- [SDO: Subspace Deconflicting Operator for Multi-Adapter Composition](https://mlsanity.com/paper/2608.13820/) (2026)

## Abstract (arXiv)

This work introduces a new approach to automatic oil painting that emphasizes the creation of dynamic and expressive brushstrokes. A pivotal challenge lies in mitigating the duplicate and common-place strokes, which often lead to less aesthetic outcomes. Inspired by the human painting process, ie, observing, comparing, and drawing, we incorporate differential image analysis into a neural oil painting model, allowing the model to effectively concentrate on the incremental impact of successive brushstrokes. To operationalize this concept, we propose the Differential Query Transformer (DQ-Transformer), a new architecture that leverages differentially derived image representations enriched with positional encoding to guide the stroke prediction process. This integration enables the model to maintain heightened sensitivity to local details, resulting in more refined and nuanced stroke generation. Furthermore, we incorporate adversarial training into our framework, enhancing the accuracy of stroke prediction and thereby improving the overall realism and fidelity of the synthesized paintings. Extensive qualitative evaluations, complemented by a controlled user study, validate that our DQ-Transformer surpasses existing methods in both visual realism and artistic authenticity, typically achieving these results with fewer strokes. The stroke-by-stroke painting animations are available on our project website.