mlsanityData to 7 Oct 2026

HiRT: Enhancing Robotic Control with Hierarchical Robot Transformers

arXiv 2410.05273 ↗First posted 12 Sep 2024cs.CV cs.AI cs.ROcited by 72

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

also in 1 more
Abstract (arXiv)

Large Vision-Language-Action (VLA) models, leveraging powerful pre trained Vision-Language Models (VLMs) backends, have shown promise in robotic control due to their impressive generalization ability. However, the success comes at a cost. Their reliance on VLM backends with billions of parameters leads to high computational costs and inference latency, limiting the testing scenarios to mainly quasi-static tasks and hindering performance in dynamic tasks requiring rapid interactions. To address these limitations, this paper proposes HiRT, a Hierarchical Robot Transformer framework that enables flexible frequency and performance trade-off. HiRT keeps VLMs running at low frequencies to capture temporarily invariant features while enabling real-time interaction through a high-frequency vision-based policy guided by the slowly updated features. Experiment results in both simulation and real-world settings demonstrate significant improvements over baseline methods. Empirically, in static tasks, we double the control frequency and achieve comparable success rates. Additionally, on novel real-world dynamic ma nipulation tasks which are challenging for previous VLA models, HiRT improves the success rate from 48% to 75%.

Key facts. Cited by 72 papers in the map (data through 7 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 56 of its 63 references that are in the map. First posted on arXiv 12 Sep 2024.

Builds on

56 of 63 references.

Closest ancestors · 7

Same direction, newest first

  1. 2024MResT: Multi-Resolution Sensing for Real-Time Control with Vision-Language Models25 Jan
  2. 2023Vision-Language Foundation Models as Effective Robot Imitators2 Nov
  3. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control28 Jul
  4. 2023EmbodiedGPT: Vision-Language Pre-Training via Embodied Chain of Thought24 May
  5. 2023PaLM-E: An Embodied Multimodal Language Model6 Mar
  6. 2023Open-World Object Manipulation using Pre-trained Vision-Language Models2 Mar
  7. 2022Visual Language Maps for Robot Navigation11 Oct

Other references · 49

  1. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
  5. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  6. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  7. 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusioncited by 2,730
  8. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  9. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,007
  10. 2022Code as Policies: Language Model Programs for Embodied Controlcited by 1,172
  11. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,168
  12. 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,057
37 more
  1. 2023VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Modelscited by 733
  2. 2022R3M: A Universal Visual Representation for Robot Manipulationcited by 678
  3. 2021CLIPort: What and Where Pathways for Robotic Manipulationcited by 617
  4. 2022ProgPrompt: Generating Situated Robot Task Plans using Large Language Modelscited by 598
  5. 2022BC-Z: Zero-Shot Task Generalization with Robotic Imitation Learningcited by 517
  6. 2022Socratic Models: Composing Zero-Shot Multimodal Reasoning with Languagecited by 513
  7. 2022Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Taskscited by 510
  8. 2023LLM+P: Empowering Large Language Models with Optimal Planning Proficiencycited by 496
  9. 2022LLM-Planner: Few-Shot Grounded Planning for Embodied Agents with Large Language Modelscited by 459
  10. 2022LM-Nav: Robotic Navigation with Large Pre-Trained Models of Language, Vision, and Actioncited by 435
  11. 2019Relay Policy Learning: Solving Long-Horizon Tasks via Imitation and Reinforcement Learningcited by 380
  12. 2023Unleashing Large-Scale Video Generative Pre-training for Visual Robot Manipulationcited by 353
  13. 2023SayPlan: Grounding Large Language Models using 3D Scene Graphs for Scalable Robot Task Planningcited by 289
  14. 2023Text2Motion: From Natural Language Instructions to Feasible Planscited by 223
  15. 2023Affordances from Human Videos as a Versatile Representation for Roboticscited by 221
  16. 2023ViNT: A Foundation Model for Visual Navigationcited by 210
  17. 2024RT-H: Action Hierarchies Using Languagecited by 204
  18. 2016Modular Multitask Reinforcement Learning with Policy Sketchescited by 200
  19. 2023LIV: Language-Image Representations and Rewards for Robotic Controlcited by 180
  20. 2023Video Language Planningcited by 160
  21. 2019A Survey of Reinforcement Learning Informed by Natural Languagecited by 154
  22. 2019Language as an Abstraction for Hierarchical Deep Reinforcement Learningcited by 138
  23. 2020Language-Conditioned Imitation Learning for Robot Manipulation Taskscited by 136
  24. 2023RT-Trajectory: Robotic Task Generalization via Hindsight Trajectory Sketchescited by 131
  25. 2023GenSim: Generating Robotic Simulation Tasks via Large Language Modelscited by 120
  26. 2021Learning Language-Conditioned Robot Behavior from Offline Data and Crowd-Sourced Annotationcited by 117
  27. 2023Toward General-Purpose Robots via Foundation Models: A Survey and Meta-Analysiscited by 99
  28. 2017Zero-Shot Task Generalization with Multi-Task Deep Reinforcement Learningcited by 97
  29. 2017Mapping Instructions and Visual Observations to Actions with Reinforcement Learningcited by 80
  30. 2021FlingBot: The Unreasonable Effectiveness of Dynamic Manipulation for Cloth Unfoldingcited by 80
  31. 2023DoReMi: Grounding Language Model by Detecting and Recovering from Plan-Execution Misalignmentcited by 58
  32. 2015Listen, Attend, and Walk: Neural Mapping of Navigational Instructions to Action Sequencescited by 50
  33. 2023Prompt a Robot to Walk with Large Language Modelscited by 44
  34. 2023SayTap: Language to Quadrupedal Locomotioncited by 40
  35. 2023Agent Instructs Large Language Models to be General Zero-Shot Reasonerscited by 32
  36. 2020PixL2R: Guiding Reinforcement Learning Using Natural Language by Mapping Pixels to Rewardscited by 30
  37. 2023Agile Catching with Whole-Body MPC and Blackbox Policy Learningcited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 38; 2026: 33.

72 papers.

Sort:

By area:

  1. 2025Ctrl-World: A Controllable Generative World Model for Robot Manipulationcited by 149
  2. 2025VLA-Adapter: An Effective Paradigm for Tiny-Scale Vision-Language-Action Modelcited by 140
  3. 2025Improving Vision-Language-Action Model with Online Reinforcement Learningcited by 107
  4. 2025UP-VLA: A Unified Understanding and Prediction Model for Embodied Agentcited by 99
  5. 2025VLA-Cache: Efficient Vision-Language-Action Manipulation via Adaptive Token Cachingcited by 83
  6. 2025OpenHelix: A Short Survey, Empirical Analysis, and Open-Source Dual-System VLA Model for Robotic Manipulationcited by 79
  7. 2025RynnVLA-002: A Unified Vision-Language-Action and World Modelcited by 70
  8. 2025Large VLM-based Vision-Language-Action Models for Robotic Manipulation: A Surveycited by 66
  9. 2024Prediction with Action: Visual Policy Learning via Joint Denoising Processcited by 64
  10. 2025Hume: Introducing System-2 Thinking in Visual-Language-Action Modelcited by 56
  11. 2025SP-VLA: A Joint Model Scheduling and Token Pruning Approach for VLA Model Accelerationcited by 56
  12. 2025Pure Vision Language Action (VLA) Models: A Comprehensive Surveycited by 46
38 more
  1. 2025LeVERB: Humanoid Whole-Body Control with Latent Vision-Language Instructioncited by 40
  2. 2026VLAW: Iterative Co-Improvement of Vision-Language-Action Policy and World Modelcited by 39
  3. 2025Vision Language Action Models in Robotic Manipulation: A Systematic Reviewcited by 38
  4. 2026VLM4VLA: Revisiting Vision-Language-Models in Vision-Language-Action Modelscited by 38
  5. 2025CO-RFT: Efficient Fine-Tuning of Vision-Language-Action Models through Chunked Offline Reinforcement Learningcited by 34
  6. 2025A Survey on Efficient Vision-Language-Action Modelscited by 34
  7. 2025Fast-in-Slow: A Dual-System Foundation Model Unifying Fast Manipulation within Slow Reasoningcited by 28
  8. 2025JARVIS-VLA: Post-Training Large-Scale Vision Language Models to Play Visual Games with Keyboards and Mousecited by 27
  9. 2025Efficient Vision-Language-Action Models for Embodied Manipulation: A Systematic Surveycited by 25
  10. 2026BagelVLA: Enhancing Long-Horizon Manipulation via Interleaved Vision-Language-Action Generationcited by 25
  11. 2025RynnVLA-001: Using Human Demonstrations to Improve Robot Manipulationcited by 24
  12. 2025Robix: A Unified Model for Robot Interaction, Reasoning and Planningcited by 23
  13. 2025Parallels Between VLA Model Post-Training and Human Motor Learning: Progress, Challenges, and Trendscited by 17
  14. 2026How Fast Can I Run My VLA? Demystifying VLA Inference Performance with VLA-Perfcited by 17
  15. 2025RoVer: Robot Reward Model as Test-Time Verifier for Vision-Language-Action Modelcited by 16
  16. 2025SwitchVLA: Execution-Aware Task Switching for Vision-Language-Action Modelscited by 15
  17. 2025MergeVLA: Cross-Skill Model Merging Toward a Generalist Vision-Language-Action Agentcited by 15
  18. 2026FAVLA: A Force-Adaptive Fast-Slow VLA model for Contact-Rich Robotic Manipulationcited by 14
  19. 2025ManualVLA: A Unified VLA Model for Chain-of-Thought Manual Generation and Robotic Manipulationcited by 13
  20. 2025Video2Act: A Dual-System Video Diffusion Policy with Robotic Spatio-Motional Modelingcited by 13
  21. 2025Reinforced Reasoning for Embodied Planningcited by 12
  22. 2025RationalVLA: A Rational Vision-Language-Action Model with Dual Systemcited by 12
  23. 2025Rethinking Latent Redundancy in Behavior Cloning: An Information Bottleneck Approach for Robot Manipulationcited by 10
  24. 2025ETA: Efficiency through Thinking Ahead, A Dual Approach to Self-Driving with Large Modelscited by 9
  25. 2025OpenHA: A Series of Open-Source Hierarchical Agentic Models in Minecraftcited by 9
  26. 2025Safe LLM-Controlled Robots with Formal Guarantees via Reachability Analysiscited by 7
  27. 2025Survey of Vision-Language-Action Models for Embodied Manipulationcited by 7
  28. 2026AsyncVLA: An Asynchronous VLA for Fast and Robust Navigation on the Edgecited by 7
  29. 2026RC-NF: Robot-Conditioned Normalizing Flow for Real-Time Anomaly Detection in Robotic Manipulationcited by 7
  30. 2025HyperVLA: Efficient Inference in Vision-Language-Action Models via Hypernetworkscited by 6
  31. 2025ELMUR: External Layer Memory with Update/Rewrite for Long-Horizon RL Problemscited by 6
  32. 2026Veo-Act: Enhancing VLA Policies with Frontier Video Modelscited by 5
  33. 2025An Anatomy of Vision-Language-Action Models: From Modules to Milestones and Challengescited by 4
  34. 2026Cortex: A Bidirectionally Aligned Embodied Agent Framework for Long-horizon Manipulationcited by 4
  35. 2026Acting While Understanding: Asynchronous Semantic-Action Decoupling for Real-Time Vision-Language-Action Modelscited by 2
  36. 2026WeaveLA: Event Driven Cross-Subtask Latent Memory Weaving for Repetitive Robot Manipulationcited by 2
  37. 2026FSD-VLN: Fast-Slow Dual-System Modeling for Aerial Long-Horizon Vision-Language Navigationcited by 2
  38. 2026Learning to Accelerate Vision-Language-Action Models through Adaptive Visual Token Cachingcited by 1

50 most cited of 72 All 72 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026What Matters in Orchestrating Robot Policies: A Systematic Study of Hierarchical VLA Agents
  2. 2025HAMSTER: Hierarchical Action Models For Open-World Robot Manipulation
  3. 2026Scaling World Model for Hierarchical Manipulation Policies
  4. 2024CogACT: A Foundational Vision-Language-Action Model for Synergizing Cognition and Action in Robotic Manipulation
  5. 2026H-VLA: Hierarchical Vision-Language-Action Model with Key-Action Reasoning and Motion Planning in a Unified Action Space
  6. 2025Long-VLA: Unleashing Long-Horizon Capability of Vision Language Action Model for Robot Manipulation
  7. 2026Beyond Flat Policies: Hierarchical Post-Training for Embodied Agents in Robotic Manipulation
  8. 2025Vision-Language-Action Models for Robotics: A Review Towards Real-World Applications