# Instructing Robots by Sketching: Learning from Demonstration via Probabilistic Diagrammatic Teaching

arXiv: https://arxiv.org/abs/2309.03835 · HTML: https://mlsanity.com/paper/2309.03835/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 34 references that are in the map.
- First posted on arXiv 7 Sep 2023.

## Builds on (most cited first, 14 of 14)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Instant Neural Graphics Primitives with a Multiresolution Hash Encoding](https://mlsanity.com/paper/2201.05989/) (2022)
- [Density estimation using Real NVP](https://mlsanity.com/paper/1605.08803/) (2016)
- [Normalizing Flows for Probabilistic Modeling and Inference](https://mlsanity.com/paper/1912.02762/) (2019)
- [Neural Sparse Voxel Fields](https://mlsanity.com/paper/2007.11571/) (2020)
- [Deep Imitation Learning for Complex Manipulation Tasks from Virtual Reality Teleoperation](https://mlsanity.com/paper/1710.04615/) (2017)
- [Analyzing Inverse Problems with Invertible Neural Networks](https://mlsanity.com/paper/1808.04730/) (2018)
- [RoboTurk: A Crowdsourcing Platform for Robotic Skill Learning through Imitation](https://mlsanity.com/paper/1811.02790/) (2018)
- [DeRF: Decomposed Radiance Fields](https://mlsanity.com/paper/2011.12490/) (2020)
- [Neural Volume Rendering: NeRF And Beyond](https://mlsanity.com/paper/2101.05204/) (2020)
- [Parallelised Diffeomorphic Sampling-based Motion Planning](https://mlsanity.com/paper/2108.11775/) (2021)
- [MotionBenchMaker: A Tool to Generate and Benchmark Motion Planning Datasets](https://mlsanity.com/paper/2112.06402/) (2021)
- [Kernel Trajectory Maps for Multi-Modal Probabilistic Motion Prediction](https://mlsanity.com/paper/1907.05127/) (2019)
- [Probabilistic Trajectory Prediction with Structural Constraints](https://mlsanity.com/paper/2107.04193/) (2021)

## Built on by (most cited first, 14 of 14)

- [Sketch-to-Skill: Bootstrapping Robot Learning with Human Drawn Trajectory Sketches](https://mlsanity.com/paper/2503.11918/) (2025)
- [VARP: Reinforcement Learning from Vision-Language Model Feedback with Agent Regularized Preferences](https://mlsanity.com/paper/2503.13817/) (2025)
- [DOSE3 : Diffusion-based Out-of-distribution detection on SE(3) trajectories](https://mlsanity.com/paper/2502.16725/) (2025)
- [GraphSeg: Segmented 3D Representations via Graph Edge Addition and Contraction](https://mlsanity.com/paper/2504.03129/) (2025)
- [Bi-Manual Joint Camera Calibration and Scene Representation](https://mlsanity.com/paper/2505.24819/) (2025)
- [PATCH: Action-Chunk-Conditioned Latent Patch Innovation Monitoring for Robot Manipulation](https://mlsanity.com/paper/2606.16690/) (2026)
- [Unifying Scene Representation and Hand-Eye Calibration with 3D Foundation Models](https://mlsanity.com/paper/2404.11683/) (2024)
- [Cross-Modal Instructions for Robot Motion Generation](https://mlsanity.com/paper/2509.21107/) (2025)
- [LineRides: Line-Guided Reinforcement Learning for Bicycle Robot Stunts](https://mlsanity.com/paper/2605.05110/) (2026)
- [TRACE: Trajectory-Routed Causal Memory for Delayed-Evidence Visuomotor Imitation](https://mlsanity.com/paper/2606.14551/) (2026)
- [Learning Generalizable 3D Manipulation With 10 Demonstrations](https://mlsanity.com/paper/2411.10203/) (2024)
- [AnyUser: Translating Sketched User Intent into Domestic Robots](https://mlsanity.com/paper/2604.04811/) (2026)
- [Tri-Manual Visuomotor Imitation Learning of Robot Policies](https://mlsanity.com/paper/2607.25731/) (2026)
- [MAVP: Map-Aware Visuomotor Policies for Mobile Manipulation](https://mlsanity.com/paper/2609.26378/) (2026)

## Abstract (arXiv)

Learning for Demonstration (LfD) enables robots to acquire new skills by imitating expert demonstrations, allowing users to communicate their instructions in an intuitive manner. Recent progress in LfD often relies on kinesthetic teaching or teleoperation as the medium for users to specify the demonstrations. Kinesthetic teaching requires physical handling of the robot, while teleoperation demands proficiency with additional hardware. This paper introduces an alternative paradigm for LfD called Diagrammatic Teaching. Diagrammatic Teaching aims to teach robots novel skills by prompting the user to sketch out demonstration trajectories on 2D images of the scene, these are then synthesised as a generative model of motion trajectories in 3D task space. Additionally, we present the Ray-tracing Probabilistic Trajectory Learning (RPTL) framework for Diagrammatic Teaching. RPTL extracts time-varying probability densities from the 2D sketches, applies ray-tracing to find corresponding regions in 3D Cartesian space, and fits a probabilistic model of motion trajectories to these regions. New motion trajectories, which mimic those sketched by the user, can then be generated from the probabilistic model. We empirically validate our framework both in simulation and on real robots, which include a fixed-base manipulator and a quadruped-mounted manipulator.