mlsanityData to 7 Oct 2026

RT-Trajectory: Robotic Task Generalization via Hindsight Trajectory Sketches

arXiv 2311.01977 ↗First posted 3 Nov 2023cs.RO cs.AIcited by 131

Not on the map. Why?

Cited by 47 in Vision-language-action models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Generalization remains one of the most important desiderata for robust robot learning systems. While recently proposed approaches show promise in generalization to novel objects, semantic concepts, or visual distribution shifts, generalization to new tasks remains challenging. For example, a language-conditioned policy trained on pick-and-place tasks will not be able to generalize to a folding task, even if the arm trajectory of folding is similar to pick-and-place. Our key insight is that this kind of generalization becomes feasible if we represent the task through rough trajectory sketches. We propose a policy conditioning method using such rough trajectory sketches, which we call RT-Trajectory, that is practical, easy to specify, and allows the policy to effectively perform new tasks that would otherwise be challenging to perform. We find that trajectory sketches strike a balance between being detailed enough to express low-level motion-centric guidance while being coarse enough to allow the learned policy to interpret the trajectory sketch in the context of situational visual observations. In addition, we show how trajectory sketches can provide a useful interface to communicate with robotic policies: they can be specified through simple human inputs like drawings or videos, or through automated methods such as modern image-generating or waypoint-generating methods. We evaluate RT-Trajectory at scale on a variety of real-world robotic tasks, and find that RT-Trajectory is able to perform a wider range of tasks compared to language-conditioned and goal-conditioned policies, when provided the same training data.

Key facts. Cited by 131 papers in the map (data through 7 Oct 2026). Builds on 29 of its 42 references that are in the map. First posted on arXiv 3 Nov 2023.

Builds on

29 of 42 references.

References · 29

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
  5. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  6. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,007
  7. 2022Code as Policies: Language Model Programs for Embodied Controlcited by 1,172
  8. 2021What Matters in Learning from Offline Human Demonstrations for Robot Manipulationcited by 699
  9. 2021Vector-quantized Image Modeling with Improved VQGANcited by 656
  10. 2021DiffusionCLIP: Text-Guided Diffusion Models for Robust Image Manipulationcited by 640
  11. 2021CLIPort: What and Where Pathways for Robotic Manipulationcited by 617
  12. 2022Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulationcited by 545
17 more
  1. 2022BC-Z: Zero-Shot Task Generalization with Robotic Imitation Learningcited by 517
  2. 2022Unified-IO: A Unified Model for Vision, Language, and Multi-Modal Taskscited by 420
  3. 2022VIMA: General Robot Manipulation with Multimodal Promptscited by 403
  4. 2019Learning Latent Plans from Playcited by 310
  5. 2017Dex-Net 2.0: Deep Learning to Plan Robust Grasps with Synthetic Point Clouds and Analytic Grasp Metricscited by 259
  6. 2023Generating Images with Multimodal Language Modelscited by 259
  7. 2021MT-Opt: Continuous Multi-Task Robotic Reinforcement Learning at Scalecited by 226
  8. 2015Supersizing Self-supervision: Learning to Grasp from 50K Tries and 700 Robot Hourscited by 208
  9. 2020Language Conditioned Imitation Learning over Unstructured Datacited by 208
  10. 2023Scaling Robot Learning with Semantically Imagined Experiencecited by 175
  11. 2021Learning Language-Conditioned Robot Behavior from Offline Data and Crowd-Sourced Annotationcited by 117
  12. 2023GenAug: Retargeting behaviors to unseen situations via Generative Augmentationcited by 103
  13. 2023Decomposing the Generalization Gap in Imitation Learning for Visual Robotic Manipulationcited by 78
  14. 2023Data Quality in Imitation Learningcited by 72
  15. 2020Human Instruction-Following with Deep Reinforcement Learning via Transfer-Learning from Textcited by 59
  16. 2020The MAGICAL Benchmark for Robust Imitationcited by 30
  17. 2023Learning Video-Conditioned Policies for Unseen Manipulation Taskscited by 15

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 5; 2024: 28; 2025: 53; 2026: 45.

131 papers.

Sort:

By area:

  1. 2024TraceVLA: Visual Trace Prompting Enhances Spatial-Temporal Awareness for Generalist Robotic Policiescited by 298
  2. 2024A Survey on Vision-Language-Action Models for Embodied AIcited by 274
  3. 2023Any-point Trajectory Modeling for Policy Learningcited by 207
  4. 2025DreamVLA: A Vision-Language-Action Model Dreamed with Comprehensive World Knowledgecited by 203
  5. 2024PIVOT: Iterative Visual Prompting Elicits Actionable Knowledge for VLMscited by 187
  6. 2024Aligning Cyber Space with Physical World: A Comprehensive Survey on Embodied AIcited by 175
  7. 2024DexCap: Scalable and Portable Mocap Data Collection System for Dexterous Manipulationcited by 149
  8. 2025Steering Your Diffusion Policy with Latent Space Reinforcement Learningcited by 149
  9. 2024Gen2Act: Human Video Generation in Novel Scenarios enables Generalizable Robot Manipulationcited by 142
  10. 2025RoboBrain: A Unified Brain Model for Robotic Manipulation from Abstract to Concretecited by 134
  11. 2024Multimodal Diffusion Transformer: Learning Versatile Behavior from Multimodal Goalscited by 131
  12. 2026π0.7: a Steerable Generalist Robotic Foundation Model with Emergent Capabilitiescited by 127
38 more
  1. 2023Toward General-Purpose Robots via Foundation Models: A Survey and Meta-Analysiscited by 99
  2. 2025DexGraspVLA: A Vision-Language-Action Framework Towards General Dexterous Graspingcited by 87
  3. 2025InternVLA-M1: A Spatially Guided Vision-Language-Action Framework for Generalist Robot Policycited by 83
  4. 2025Vision-Language-Action Models for Robotics: A Review Towards Real-World Applicationscited by 82
  5. 2024HiRT: Enhancing Robotic Control with Hierarchical Robot Transformerscited by 72
  6. 2025InstructVLA: Vision-Language-Action Instruction Tuning from Understanding to Manipulationcited by 71
  7. 2025SafeVLA: Towards Safety Alignment of Vision-Language-Action Model via Constrained Learningcited by 69
  8. 2025Large VLM-based Vision-Language-Action Models for Robotic Manipulation: A Surveycited by 66
  9. 2024Embodied Understanding of Driving Scenarioscited by 63
  10. 2024Real-World Robot Applications of Foundation Models: A Reviewcited by 60
  11. 2024Diffusion-VLA: Generalizable and Interpretable Robot Foundation Model via Self-Generated Reasoningcited by 57
  12. 2023QUAR-VLA: Vision-Language-Action Model for Quadruped Robotscited by 55
  13. 2025Training Strategies for Efficient Embodied Reasoningcited by 53
  14. 2025RoboGround: Robotic Manipulation with Grounded Vision-Language Priorscited by 50
  15. 2024Efficient Diffusion Transformer Policies with Mixture of Expert Denoisers for Multitask Learningcited by 46
  16. 2024RT-Sketch: Goal-Conditioned Imitation Learning from Hand-Drawn Sketchescited by 41
  17. 2024RT-Affordance: Affordances are Versatile Intermediate Representations for Robot Manipulationcited by 40
  18. 2025CoMo: Learning Continuous Latent Motion from Internet Videos for Scalable Robot Learningcited by 37
  19. 2025Embodied-R1: Reinforced Embodied Reasoning for General Robotic Manipulationcited by 36
  20. 2024Inference-Time Policy Steering through Human Interactionscited by 35
  21. 2025VLA-OS: Structuring and Dissecting Planning Representations and Paradigms in Vision-Language-Action Modelscited by 34
  22. 2025Generating Robot Constitutions & Benchmarks for Semantic Safetycited by 28
  23. 2025RoboOS: A Hierarchical Embodied Framework for Cross-Embodiment and Multi-Agent Collaborationcited by 28
  24. 2025Pre-training Auto-regressive Robotic Models with 4D Representationscited by 27
  25. 2025Robotic Visual Instructioncited by 21
  26. 2025Enhancing Generalization in Vision-Language-Action Models by Preserving Pretrained Representationscited by 21
  27. 2024RH20T-P: A Primitive-Level Robotic Dataset Towards Composable Generalization Agentscited by 20
  28. 2024Affordance-Guided Reinforcement Learning via Visual Promptingcited by 20
  29. 2024ROCKET-1: Mastering Open-World Interaction with Visual-Temporal Context Promptingcited by 20
  30. 2026PALM: Progress-Aware Policy Learning via Affordance Reasoning for Long-Horizon Robotic Manipulationcited by 20
  31. 2026Steerable Vision-Language-Action Policies for Embodied Reasoning and Hierarchical Controlcited by 19
  32. 2024Generative Image as Action Modelscited by 18
  33. 2025See, Point, Fly: A Learning-Free VLM Framework for Universal Unmanned Aerial Navigationcited by 18
  34. 2025Seeing to Act, Prompting to Specify: A Bayesian Factorization of Vision Language Action Policycited by 17
  35. 2026How Fast Can I Run My VLA? Demystifying VLA Inference Performance with VLA-Perfcited by 17
  36. 2026VP-VLA: Visual Prompting as an Interface for Vision-Language-Action Modelscited by 17
  37. 2023Harmonic Mobile Manipulationcited by 16
  38. 2025Diffusion Trajectory-guided Policy for Long-horizon Robot Manipulationcited by 15

50 most cited of 131 All 131 (JSON)