mlsanityData to 7 Oct 2026

RT-H: Action Hierarchies Using Language

arXiv 2403.01823 ↗First posted 4 Mar 2024cs.RO cs.AIcited by 204

Not on the map. Why?

Cited by 95 in Vision-language-action models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Language provides a way to break down complex concepts into digestible pieces. Recent works in robot imitation learning use language-conditioned policies that predict actions given visual observations and the high-level task specified in language. These methods leverage the structure of natural language to share data between semantically similar tasks (e.g., "pick coke can" and "pick an apple") in multi-task datasets. However, as tasks become more semantically diverse (e.g., "pick coke can" and "pour cup"), sharing data between tasks becomes harder, so learning to map high-level tasks to actions requires much more demonstration data. To bridge tasks and actions, our insight is to teach the robot the language of actions, describing low-level motions with more fine-grained phrases like "move arm forward". Predicting these language motions as an intermediate step between tasks and actions forces the policy to learn the shared structure of low-level motions across seemingly disparate tasks. Furthermore, a policy that is conditioned on language motions can easily be corrected during execution through human-specified language motions. This enables a new paradigm for flexible policies that can learn from human intervention in language. Our method RT-H builds an action hierarchy using language motions: it first learns to predict language motions, and conditioned on this and the high-level task, it predicts actions, using visual context at all stages. We show that RT-H leverages this language-action hierarchy to learn policies that are more robust and flexible by effectively tapping into multi-task datasets. We show that these policies not only allow for responding to language interventions, but can also learn from such interventions and outperform methods that learn from teleoperated interventions. Our website and videos are found at https://rt-hierarchy.github.io.

Key facts. Cited by 204 papers in the map (data through 7 Oct 2026). Builds on 46 of its 58 references that are in the map. First posted on arXiv 4 Mar 2024.

Builds on

46 of 58 references.

References · 46

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
  3. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  4. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  5. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,007
  6. 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,057
  7. 2021What Matters in Learning from Offline Human Demonstrations for Robot Manipulationcited by 699
  8. 2022R3M: A Universal Visual Representation for Robot Manipulationcited by 678
  9. 2021CLIPort: What and Where Pathways for Robotic Manipulationcited by 617
  10. 2022BC-Z: Zero-Shot Task Generalization with Robotic Imitation Learningcited by 517
  11. 2022VIP: Towards Universal Visual Reward and Representation via Value-Implicit Pre-Trainingcited by 384
  12. 2019Learning Latent Plans from Playcited by 310
34 more
  1. 2023PaLI-X: On Scaling up a Multilingual Vision and Language Modelcited by 254
  2. 2018HG-DAgger: Interactive Imitation Learning with Human Expertscited by 245
  3. 2022Interactive Language: Talking to Robots in Real Timecited by 217
  4. 2024PIVOT: Iterative Visual Prompting Elicits Actionable Knowledge for VLMscited by 187
  5. 2023Language-Driven Representation Learning for Roboticscited by 163
  6. 2022What Matters in Language Conditioned Robotic Imitation Learning over Unstructured Datacited by 137
  7. 2020Language-Conditioned Imitation Learning for Robot Manipulation Taskscited by 136
  8. 2023RoboVQA: Multimodal Long-Horizon Reasoning for Roboticscited by 116
  9. 2020Human-in-the-Loop Imitation Learning using Remote Teleoperationcited by 103
  10. 2021Skill Induction and Planning with Latent Languagecited by 99
  11. 2021ThriftyDAgger: Budget-Aware Novelty and Risk Gating for Interactive Imitation Learningcited by 98
  12. 2018EnsembleDAgger: A Bayesian Approach to Safe Imitation Learningcited by 96
  13. 2021Coarse-to-Fine Imitation Learning: Robot Manipulation from a Single Demonstrationcited by 92
  14. 2017Robust Imitation of Diverse Behaviorscited by 85
  15. 2021Bottom-Up Skill Discovery from Unsegmented Demonstrations for Long-Horizon Robot Manipulationcited by 82
  16. 2023Data Quality in Imitation Learningcited by 72
  17. 2022Correcting Robot Plans with Natural Language Feedbackcited by 68
  18. 2023"No, to the Right" -- Online Language Corrections for Robotic Manipulation via Shared Autonomycited by 60
  19. 2020Learning Robot Skills with Temporal Variational Inferencecited by 54
  20. 2023HYDRA: Hybrid Robot Actions for Imitation Learningcited by 54
  21. 2021LazyDAgger: Reducing Context Switching in Interactive Imitation Learningcited by 50
  22. 2023Distilling and Retrieving Generalizable Knowledge for Robot Manipulation via Language Correctionscited by 44
  23. 2017DDCO: Discovery of Deep Continuous Options for Robot Learning from Demonstrationscited by 43
  24. 2021ELLA: Exploration through Learned Language Abstractioncited by 42
  25. 2023KITE: Keypoint-Conditioned Policies for Semantic Manipulationcited by 39
  26. 2021Hierarchical Few-Shot Imitation with Skill Transition Modelscited by 37
  27. 2018Guiding Policies with Language via Meta-Learningcited by 34
  28. 2023Thought Cloning: Learning to Think while Acting by Imitating Human Thinkingcited by 30
  29. 2021SKID RAW: Skill Discovery from Raw Trajectoriescited by 22
  30. 2021Learning Human Objectives from Sequences of Physical Correctionscited by 21
  31. 2021Physical Interaction as Communication: Learning Robot Objectives Online from Human Correctionscited by 19
  32. 2022Reshaping Robot Trajectories Using Natural Language Commands: A Study of Multi-Modal Data Alignment Using Transformerscited by 17
  33. 2022PLATO: Predicting Latent Affordances Through Object-Centric Playcited by 12
  34. 2023Improving Long-Horizon Imitation Through Instruction Predictioncited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 40; 2025: 91; 2026: 73.

204 papers.

Sort:

By area:

  1. 2025FAST: Efficient Action Tokenization for Vision-Language-Action Modelscited by 643
  2. 2024Latent Action Pretraining from Videoscited by 302
  3. 2024A Survey on Vision-Language-Action Models for Embodied AIcited by 274
  4. 2025Hi Robot: Open-Ended Instruction Following with Hierarchical Vision-Language-Action Modelscited by 216
  5. 2025EmbodiedBench: Comprehensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agentscited by 213
  6. 2025DreamVLA: A Vision-Language-Action Model Dreamed with Comprehensive World Knowledgecited by 203
  7. 2025DexVLA: Vision-Language Model with Plug-In Diffusion Expert for General Robot Controlcited by 191
  8. 2025HybridVLA: Collaborative Diffusion and Autoregression in a Unified Vision-Language-Action Modelcited by 176
  9. 2024Aligning Cyber Space with Physical World: A Comprehensive Survey on Embodied AIcited by 175
  10. 2024RoboMIND: Benchmark on Multi-embodiment Intelligence Normative Data for Robot Manipulationcited by 166
  11. 2025Cosmos-Reason1: From Physical Common Sense To Embodied Reasoningcited by 159
  12. 2025RoboBrain: A Unified Brain Model for Robotic Manipulation from Abstract to Concretecited by 134
38 more
  1. 2025Unified Vision-Language-Action Modelcited by 128
  2. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 125
  3. 2024Yell At Your Robot: Improving On-the-Fly from Language Correctionscited by 123
  4. 2025OneTwoVLA: A Unified Vision-Language-Action Model with Adaptive Reasoningcited by 105
  5. 2026VLA-JEPA: Enhancing Vision-Language-Action Model with Latent World Modelcited by 104
  6. 2025A Survey on Vision-Language-Action Models: An Action Tokenization Perspectivecited by 96
  7. 2025InternVLA-M1: A Spatially Guided Vision-Language-Action Framework for Generalist Robot Policycited by 83
  8. 2025Vision-Language-Action Models for Robotics: A Review Towards Real-World Applicationscited by 82
  9. 2024OAKINK2: A Dataset of Bimanual Hands-Object Manipulation in Complex Task Completioncited by 78
  10. 2025Dita: Scaling Diffusion Transformer for Generalist Vision-Language-Action Policycited by 73
  11. 2024HiRT: Enhancing Robotic Control with Hierarchical Robot Transformerscited by 72
  12. 2025Vision-Language-Action (VLA) Models: Concepts, Progress, Applications and Challengescited by 72
  13. 2025InstructVLA: Vision-Language-Action Instruction Tuning from Understanding to Manipulationcited by 71
  14. 2025CogVLA: Cognition-Aligned Vision-Language-Action Model via Instruction-Driven Routing & Sparsificationcited by 70
  15. 2025SafeVLA: Towards Safety Alignment of Vision-Language-Action Model via Constrained Learningcited by 69
  16. 2025Large VLM-based Vision-Language-Action Models for Robotic Manipulation: A Surveycited by 66
  17. 2024Diffusion-VLA: Generalizable and Interpretable Robot Foundation Model via Self-Generated Reasoningcited by 57
  18. 2025SP-VLA: A Joint Model Scheduling and Token Pruning Approach for VLA Model Accelerationcited by 56
  19. 2025RoboGround: Robotic Manipulation with Grounded Vision-Language Priorscited by 50
  20. 2025VLAS: Vision-Language-Action Model With Speech Instructions For Customized Robot Manipulationcited by 48
  21. 2024RACER: Rich Language-Guided Failure Recovery Policies for Imitation Learningcited by 46
  22. 2025SoFar: Language-Grounded Orientation Bridges Spatial Reasoning and Object Manipulationcited by 46
  23. 2024Emma-X: An Embodied Multimodal Action Model with Grounded Chain of Thought and Look-ahead Spatial Reasoningcited by 45
  24. 2024STRAP: Robot Sub-Trajectory Retrieval for Augmented Policy Learningcited by 42
  25. 2024RT-Affordance: Affordances are Versatile Intermediate Representations for Robot Manipulationcited by 40
  26. 2025RoboFAC: A Comprehensive Framework for Robotic Failure Analysis and Correctioncited by 37
  27. 2025Multi-agent Embodied AI: Advances and Future Directionscited by 36
  28. 2026Do World Action Models Generalize Better than VLAs? A Robustness Studycited by 35
  29. 2025VLA-OS: Structuring and Dissecting Planning Representations and Paradigms in Vision-Language-Action Modelscited by 34
  30. 2024FlowRetrieval: Flow-Guided Data Retrieval for Few-Shot Imitation Learningcited by 32
  31. 2025ZeroMimic: Distilling Robotic Manipulation Skills from Web Videoscited by 31
  32. 2025Shortcut Learning in Generalist Robot Policies: The Role of Dataset Diversity and Fragmentationcited by 31
  33. 2025LLaDA-VLA: Vision Language Diffusion Action Modelscited by 30
  34. 2025PAI-Bench: A Comprehensive Benchmark For Physical AIcited by 30
  35. 2025Fast-in-Slow: A Dual-System Foundation Model Unifying Fast Manipulation within Slow Reasoningcited by 28
  36. 2026LaST0: Latent Spatio-Temporal Chain-of-Thought for Robotic Vision-Language-Action Modelcited by 28
  37. 2025JARVIS-VLA: Post-Training Large-Scale Vision Language Models to Play Visual Games with Keyboards and Mousecited by 27
  38. 2026StarVLA-α: Reducing Complexity in Vision-Language-Action Systemscited by 27

50 most cited of 204 All 204 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2020Language Conditioned Imitation Learning over Unstructured Data
  2. 2023Language-Conditioned Imitation Learning with Base Skill Priors under Unstructured Data
  3. 2024Rethinking Mutual Information for Language Conditioned Skill Discovery on Imitation Learning
  4. 2023Bridging Language and Action: A Survey of Language-Conditioned Robot Manipulation
  5. 2024Video-Language Critic: Transferable Reward Functions for Language-Conditioned Robotics
  6. 2023Goal Representations for Instruction Following: A Semi-Supervised Language Interface to Control
  7. 2026Language-Critique Imitation Learning from Suboptimal Demonstrations
  8. 2021CALVIN: A Benchmark for Language-Conditioned Policy Learning for Long-Horizon Robot Manipulation Tasks