# Sketch-to-Skill: Bootstrapping Robot Learning with Human Drawn Trajectory Sketches

arXiv: https://arxiv.org/abs/2503.11918 · HTML: https://mlsanity.com/paper/2503.11918/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Placed in Robot manipulation imitation learning, one of 13 directions in Offline RL and imitation learning from data.
- Builds on 18 of its 41 references that are in the map.
- First posted on arXiv 14 Mar 2025.

## Placed in

- Offline RL and imitation learning from data / [Robot manipulation imitation learning](https://mlsanity.com/offlinerl/robot-manipulation-imitation-learning/)

## Builds on (most cited first, 18 of 18)

- [Addressing Function Approximation Error in Actor-Critic Methods](https://mlsanity.com/paper/1802.09477/) (2018)
- [Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learning](https://mlsanity.com/paper/1910.10897/) (2019)
- [What Matters in Learning from Offline Human Demonstrations for Robot Manipulation](https://mlsanity.com/paper/2108.03298/) (2021)
- [Deep Imitation Learning for Complex Manipulation Tasks from Virtual Reality Teleoperation](https://mlsanity.com/paper/1710.04615/) (2017)
- [Masked World Models for Visual Control](https://mlsanity.com/paper/2206.14244/) (2022)
- [RT-Trajectory: Robotic Task Generalization via Hindsight Trajectory Sketches](https://mlsanity.com/paper/2311.01977/) (2023)
- [On Bringing Robots Home](https://mlsanity.com/paper/2311.16098/) (2023)
- [A Walk in the Park: Learning to Walk in 20 Minutes With Model-Free Reinforcement Learning](https://mlsanity.com/paper/2208.07860/) (2022)
- [Imitation Bootstrapped Reinforcement Learning](https://mlsanity.com/paper/2311.02198/) (2023)
- [RT-Sketch: Goal-Conditioned Imitation Learning from Hand-Drawn Sketches](https://mlsanity.com/paper/2403.02709/) (2024)
- [SceneTrilogy: On Human Scene-Sketch and its Complementarity with Photo and Text](https://mlsanity.com/paper/2204.11964/) (2022)
- [What Can Human Sketches Do for Object Detection?](https://mlsanity.com/paper/2303.15149/) (2023)
- [Sketch2Saliency: Learning to Detect Salient Objects from Human Drawings](https://mlsanity.com/paper/2303.11502/) (2023)
- [Instructing Robots by Sketching: Learning from Demonstration via Probabilistic Diagrammatic Teaching](https://mlsanity.com/paper/2309.03835/) (2023)
- [Learning Variable Compliance Control From a Few Demonstrations for Bimanual Robot with Haptic Feedback Teleoperation System](https://mlsanity.com/paper/2406.14990/) (2024)
- [A Comparison of Imitation Learning Algorithms for Bimanual Manipulation](https://mlsanity.com/paper/2408.06536/) (2024)
- [PLANRL: A Motion Planning and Imitation Learning Framework to Bootstrap Reinforcement Learning](https://mlsanity.com/paper/2408.04054/) (2024)
- [Beyond Joint Demonstrations: Personalized Expert Guidance for Efficient Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2403.08936/) (2024)

## Built on by (most cited first, 9 of 9)

- [TraceGen: World Modeling in 3D Trace Space Enables Learning from Cross-Embodiment Videos](https://mlsanity.com/paper/2511.21690/) (2025)
- [VARP: Reinforcement Learning from Vision-Language Model Feedback with Agent Regularized Preferences](https://mlsanity.com/paper/2503.13817/) (2025)
- [AFFORD2ACT: Affordance-Guided Automatic Keypoint Selection for Generalizable and Lightweight Robotic Manipulation](https://mlsanity.com/paper/2510.01433/) (2025)
- [Cross-Modal Instructions for Robot Motion Generation](https://mlsanity.com/paper/2509.21107/) (2025)
- [SketchPlan: Diffusion Based Drone Planning From Human Sketches](https://mlsanity.com/paper/2510.03545/) (2025)
- [PRISM: Performer RS-IMLE for Single-pass Multisensory Imitation Learning](https://mlsanity.com/paper/2602.02396/) (2026)
- [LineRides: Line-Guided Reinforcement Learning for Bicycle Robot Stunts](https://mlsanity.com/paper/2605.05110/) (2026)
- [AnyUser: Translating Sketched User Intent into Domestic Robots](https://mlsanity.com/paper/2604.04811/) (2026)
- [3DWay: Generalizing Robot Manipulation via 3D Consistent Waypoints](https://mlsanity.com/paper/2609.08224/) (2026)

## Abstract (arXiv)

Training robotic manipulation policies traditionally requires numerous demonstrations and/or environmental rollouts. While recent Imitation Learning (IL) and Reinforcement Learning (RL) methods have reduced the number of required demonstrations, they still rely on expert knowledge to collect high-quality data, limiting scalability and accessibility. We propose Sketch-to-Skill, a novel framework that leverages human-drawn 2D sketch trajectories to bootstrap and guide RL for robotic manipulation. Our approach extends beyond previous sketch-based methods, which were primarily focused on imitation learning or policy conditioning, limited to specific trained tasks. Sketch-to-Skill employs a Sketch-to-3D Trajectory Generator that translates 2D sketches into 3D trajectories, which are then used to autonomously collect initial demonstrations. We utilize these sketch-generated demonstrations in two ways: to pre-train an initial policy through behavior cloning and to refine this policy through RL with guided exploration. Experimental results demonstrate that Sketch-to-Skill achieves ~96% of the performance of the baseline model that leverages teleoperated demonstration data, while exceeding the performance of a pure reinforcement learning policy by ~170%, only from sketch inputs. This makes robotic manipulation learning more accessible and potentially broadens its applications across various domains.