mlsanityData to 14 Oct 2026

RT-Sketch: Goal-Conditioned Imitation Learning from Hand-Drawn Sketches

arXiv 2403.02709 ↗First posted 5 Mar 2024cs.ROcited by 41

On the map: Offline RL and imitation learning from data → Robot manipulation imitation learning

Abstract (arXiv)

Natural language and images are commonly used as goal representations in goal-conditioned imitation learning (IL). However, natural language can be ambiguous and images can be over-specified. In this work, we propose hand-drawn sketches as a modality for goal specification in visual imitation learning. Sketches are easy for users to provide on the fly like language, but similar to images they can also help a downstream policy to be spatially-aware and even go beyond images to disambiguate task-relevant from task-irrelevant objects. We present RT-Sketch, a goal-conditioned policy for manipulation that takes a hand-drawn sketch of the desired scene as input, and outputs actions. We train RT-Sketch on a dataset of paired trajectories and corresponding synthetically generated goal sketches. We evaluate this approach on six manipulation skills involving tabletop object rearrangements on an articulated countertop. Experimentally we find that RT-Sketch is able to perform on a similar level to image or language-conditioned agents in straightforward settings, while achieving greater robustness when language goals are ambiguous or visual distractors are present. Additionally, we show that RT-Sketch has the capacity to interpret and act upon sketches with varied levels of specificity, ranging from minimal line drawings to detailed, colored drawings. For supplementary material and videos, please refer to our website: http://rt-sketch.github.io.

Key facts. Cited by 41 papers in the map (data through 14 Oct 2026). Placed in Robot manipulation imitation learning, one of 13 directions in Offline RL and imitation learning from data. Builds on 37 of its 49 references that are in the map. First posted on arXiv 5 Mar 2024.

Builds on

37 of 49 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023Data Quality in Imitation Learning4 Jun
  2. 2022BC-Z: Zero-Shot Task Generalization with Robotic Imitation Learning4 Feb

Other references · 35

  1. 2017Attention Is All You Needcited by 60,361
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  4. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,442
  5. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,503
  6. 2016Image-to-Image Translation with Conditional Adversarial Networkscited by 4,527
  7. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 3,080
  8. 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,611
  9. 2017FiLM: Visual Reasoning with a General Conditioning Layercited by 2,333
  10. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,061
  11. 2017Hindsight Experience Replaycited by 1,136
  12. 2018Behavioral Cloning from Observationcited by 492
23 more
  1. 2022VIMA: General Robot Manipulation with Multimodal Promptscited by 407
  2. 2023Goal-Conditioned Imitation Learning using Score-based Diffusion Policiescited by 236
  3. 2022Interactive Language: Talking to Robots in Real Timecited by 218
  4. 2020Language Conditioned Imitation Learning over Unstructured Datacited by 209
  5. 2016Modular Multitask Reinforcement Learning with Policy Sketchescited by 203
  6. 2023Language-Driven Representation Learning for Roboticscited by 165
  7. 2019Goal-conditioned Imitation Learningcited by 162
  8. 2019kPAM: KeyPoint Affordances for Category-Level Robotic Manipulationcited by 153
  9. 2023RT-Trajectory: Robotic Task Generalization via Hindsight Trajectory Sketchescited by 133
  10. 2022Can Foundation Models Perform Zero-Shot Task Specification For Robot Manipulation?cited by 70
  11. 2019Photo-Sketching: Inferring Contour Drawings from Imagescited by 68
  12. 2022CLIPascene: Scene Sketching with Different Types and Levels of Abstractioncited by 64
  13. 2023"No, to the Right" -- Online Language Corrections for Robotic Manipulation via Shared Autonomycited by 60
  14. 2019Form2Fit: Learning Shape Priors for Generalizable Assembly from Disassemblycited by 51
  15. 2023Picture that Sketch: Photorealistic Image Generation from Abstract Sketchescited by 34
  16. 2019Mechanical Search: Multi-Step Retrieval of a Target Object Occluded by Cluttercited by 30
  17. 2022SceneTrilogy: On Human Scene-Sketch and its Complementarity with Photo and Textcited by 30
  18. 2022Doodle It Yourself: Class Incremental Learning by Drawing a Few Sketchescited by 23
  19. 2023What Can Human Sketches Do for Object Detection?cited by 19
  20. 2023Sketch2Saliency: Learning to Detect Salient Objects from Human Drawingscited by 17
  21. 2021Emergent Graphical Conventions in a Visual Communication Gamecited by 16
  22. 2023Sketching Robot Programs On the Flycited by 4
  23. 2023Emergent Communication in Interactive Sketch Question Answeringcited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 10; 2025: 19; 2026: 12.

41 papers.

Sort:

By area:

  1. 2025Vision-Language-Action Models for Robotics: A Review Towards Real-World Applicationscited by 84
  2. 2025Large VLM-based Vision-Language-Action Models for Robotic Manipulation: A Surveycited by 69
  3. 2024RAM: Retrieval-Based Affordance Transfer for Generalizable Zero-Shot Robotic Manipulationcited by 58
  4. 2025RoboGround: Robotic Manipulation with Grounded Vision-Language Priorscited by 51
  5. 2025Interleave-VLA: Enhancing Robot Manipulation with Interleaved Image-Text Instructionscited by 51
  6. 2024RT-Affordance: Affordances are Versatile Intermediate Representations for Robot Manipulationcited by 43
  7. 2024Inference-Time Policy Steering through Human Interactionscited by 38
  8. 2025ContextVLA: Vision-Language-Action Model with Amortized Multi-Frame Contextcited by 35
  9. 2025Learning Long-Context Diffusion Policies via Past-Token Predictioncited by 30
  10. 2024TidyBot++: An Open-Source Holonomic Mobile Manipulator for Robot Learningcited by 22
  11. 2025Robotic Visual Instructioncited by 22
  12. 2024ROCKET-1: Mastering Open-World Interaction with Visual-Temporal Context Promptingcited by 20
29 more
  1. 2024Generative Image as Action Modelscited by 18
  2. 2025Manual2Skill: Learning to Read Manuals and Acquire Robotic Skills for Furniture Assembly Using Vision-Language Modelscited by 14
  3. 2025Diagnose, Correct, and Learn from Manipulation Failures via Visual Symbolscited by 14
  4. 2025Sketch-to-Skill: Bootstrapping Robot Learning with Human Drawn Trajectory Sketchescited by 9
  5. 2026LiLa-WAM: Lightweight Latent Reasoning World-Action Model for Robotic Manipulationcited by 9
  6. 2024Helpful DoggyBot: Open-World Object Fetching using Legged Robots and Vision-Language Modelscited by 8
  7. 2025ROCKET-2: Steering Visuomotor Policy via Cross-View Goal Alignmentcited by 7
  8. 2025Chain-of-Modality: Learning Manipulation Programs from Multimodal Human Videos with Vision-Language-Modelscited by 7
  9. 2025CrayonRobo: Object-Centric Prompt-Driven Vision-Language-Action Model for Robotic Manipulationcited by 6
  10. 2024BiKC: Keypose-Conditioned Consistency Policy for Bimanual Robotic Manipulationcited by 5
  11. 2025Behavioral Exploration: Learning to Explore via In-Context Adaptationcited by 5
  12. 2026Action-Sketcher: From Reasoning to Action via Visual Sketches for Long-Horizon Robotic Manipulationcited by 4
  13. 2026Contact-Anchored Policies: Contact Conditioning Creates Strong Robot Utility Modelscited by 4
  14. 2026Look Before Acting: Enhancing Vision Foundation Representations for Vision-Language-Action Modelscited by 4
  15. 2024Object-centric proto-symbolic behavioural reasoning from pixelscited by 2
  16. 2025FoldNet: Learning Generalizable Closed-Loop Policy for Garment Folding via Keypoint-Driven Asset and Demonstration Synthesiscited by 1
  17. 2025Cross-Modal Instructions for Robot Motion Generationcited by 1
  18. 2025SketchPlan: Diffusion Based Drone Planning From Human Sketchescited by 1
  19. 2025Toward Autonomous Soft Robotic Endovascular Navigation via Imitation Learningcited by 1
  20. 2025Correspondence-Oriented Imitation Learning: Flexible Visuomotor Control with 3D Conditioningcited by 1
  21. 2026GuidedAttention: Interpretable and Correctable Visual Attention for OOD-Robust Robot Manipulation via Imitation Learningcited by 1
  22. 2026Cross-Embodiment Transfer via Behavior-Aligned Representationscited by 1
  23. 2024One-Shot Robust Imitation Learning for Long-Horizon Visuomotor Tasks from Unsegmented Demonstrationscited by 0
  24. 2026Visual Prompt Guided Unified Pushing Policycited by 0
  25. 2026Learning When to Cooperate Under Heterogeneous Goalscited by 0
  26. 2026AnyUser: Translating Sketched User Intent into Domestic Robotscited by 0
  27. 2026Flow Control: Steering Vision-Language-Action Models with Simple Real-Time Inputscited by 0
  28. 2026LIBERO-VIFO: Benchmarking the Capability and Safety of Visual Cue Following in Vision-Language-Action Modelscited by 0
  29. 2026NarrativeFlow: Flow-Based Vision-Language-Action Model Using Robot Velocity Fieldscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2020Language-Conditioned Imitation Learning for Robot Manipulation Tasks
  2. 2026Decoupling Semantics and Geometric Grounding: Spatial Visual Prompts for Language-Conditioned Imitation Learning
  3. 2025L2D2: Robot Learning from 2D Drawings
  4. 2019Motion Reasoning for Goal-Based Imitation Learning
  5. 2024Keypoint Action Tokens Enable In-Context Imitation Learning in Robotics
  6. 2026Spatially Conditioned Diffusion Policy: Learning Precise and Robust Manipulation with a Single RGB Camera
  7. 2021Manipulator-Independent Representations for Visual Imitation
  8. 2021The Surprising Effectiveness of Representation Learning for Visual Imitation