# From Scratch to Sketch: Deep Decoupled Hierarchical Reinforcement Learning for Robotic Sketching Agent

arXiv: https://arxiv.org/abs/2208.04833 · HTML: https://mlsanity.com/paper/2208.04833/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 32 references that are in the map.
- First posted on arXiv 9 Aug 2022.

## Builds on (most cited first, 17 of 17)

- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Spectral Normalization for Generative Adversarial Networks](https://mlsanity.com/paper/1802.05957/) (2018)
- [Deep Reinforcement Learning for Autonomous Driving: A Survey](https://mlsanity.com/paper/2002.00444/) (2020)
- [Data-Efficient Hierarchical Reinforcement Learning](https://mlsanity.com/paper/1805.08296/) (2018)
- [Sim-to-Real Transfer in Deep Reinforcement Learning for Robotics: a Survey](https://mlsanity.com/paper/2009.13303/) (2020)
- [A Neural Representation of Sketch Drawings](https://mlsanity.com/paper/1704.03477/) (2017)
- [Visual Reinforcement Learning with Imagined Goals](https://mlsanity.com/paper/1807.04742/) (2018)
- [Planning with Goal-Conditioned Policies](https://mlsanity.com/paper/1911.08453/) (2019)
- [Unsupervised Control Through Non-Parametric Discriminative Rewards](https://mlsanity.com/paper/1811.11359/) (2018)
- [Synthesizing Programs for Images using Reinforced Adversarial Learning](https://mlsanity.com/paper/1804.01118/) (2018)
- [Learning to Paint With Model-based Deep Reinforcement Learning](https://mlsanity.com/paper/1903.04411/) (2019)
- [Why Does Hierarchy (Sometimes) Work So Well in Reinforcement Learning?](https://mlsanity.com/paper/1909.10618/) (2019)
- [Many-Goals Reinforcement Learning](https://mlsanity.com/paper/1806.09605/) (2018)
- [Unsupervised Doodling and Painting with Improved SPIRAL](https://mlsanity.com/paper/1910.01007/) (2019)
- [Learning Multi-Level Hierarchies with Hindsight](https://mlsanity.com/paper/1712.00948/) (2017)
- [Content Masked Loss: Human-Like Brush Stroke Planning in a Reinforcement Learning Painting Agent](https://mlsanity.com/paper/2012.10043/) (2020)
- [Artistic Style in Robotic Painting; a Machine Learning Approach to Learning Brushstroke from Human Artists](https://mlsanity.com/paper/2007.03647/) (2020)

## Built on by (most cited first, 2 of 2)

- [Neural Collage Transfer: Artistic Reconstruction via Material Manipulation](https://mlsanity.com/paper/2311.02202/) (2023)
- [IMPASTO: Integrating Model-Based Planning with Learned Dynamics Models for Robotic Oil Painting Reproduction](https://mlsanity.com/paper/2603.29315/) (2026)

## Abstract (arXiv)

We present an automated learning framework for a robotic sketching agent that is capable of learning stroke-based rendering and motor control simultaneously. We formulate the robotic sketching problem as a deep decoupled hierarchical reinforcement learning; two policies for stroke-based rendering and motor control are learned independently to achieve sub-tasks for drawing, and form a hierarchy when cooperating for real-world drawing. Without hand-crafted features, drawing sequences or trajectories, and inverse kinematics, the proposed method trains the robotic sketching agent from scratch. We performed experiments with a 6-DoF robot arm with 2F gripper to sketch doodles. Our experimental results show that the two policies successfully learned the sub-tasks and collaborated to sketch the target images. Also, the robustness and flexibility were examined by varying drawing tools and surfaces.