mlsanityData to 14 Oct 2026

Hi Robot: Open-Ended Instruction Following with Hierarchical Vision-Language-Action Models

arXiv 2502.19417 ↗First posted 26 Feb 2025cs.RO cs.AI cs.LGcited by 221

Not on the map. Why?

Cited by 120 in Vision-language-action models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Generalist robots that can perform a range of different tasks in open-world settings must be able to not only reason about the steps needed to accomplish their goals, but also process complex instructions, prompts, and even feedback during task execution. Intricate instructions (e.g., "Could you make me a vegetarian sandwich?" or "I don't like that one") require not just the ability to physically perform the individual steps, but the ability to situate complex commands and feedback in the physical world. In this work, we describe a system that uses vision-language models in a hierarchical structure, first reasoning over complex prompts and user feedback to deduce the most appropriate next step to fulfill the task, and then performing that step with low-level actions. In contrast to direct instruction following methods that can fulfill simple commands ("pick up the cup"), our system can reason through complex prompts and incorporate situated feedback during task execution ("that's not trash"). We evaluate our system across three robotic platforms, including single-arm, dual-arm, and dual-arm mobile robots, demonstrating its ability to handle tasks such as cleaning messy tables, making sandwiches, and grocery shopping. Videos are available at https://www.pi.website/research/hirobot

Key facts. Cited by 221 papers in the map (data through 14 Oct 2026). Builds on 40 of its 51 references that are in the map. First posted on arXiv 26 Feb 2025.

Builds on

40 of 51 references.

References · 40

  1. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  2. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 3,080
  3. 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,887
  4. 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusioncited by 2,838
  5. 2024π0: A Vision-Language-Action Flow Model for General Robot Controlcited by 2,407
  6. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,348
  7. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,071
  8. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,061
  9. 2023Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardwarecited by 1,568
  10. 2024Octo: An Open-Source Generalist Robot Policycited by 1,484
  11. 2022Code as Policies: Language Model Programs for Embodied Controlcited by 1,210
  12. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,035
28 more
  1. 2023VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Modelscited by 750
  2. 2024PaliGemma: A versatile 3B VLM for transfercited by 696
  3. 2024RDT-1B: a Diffusion Foundation Model for Bimanual Manipulationcited by 691
  4. 2025FAST: Efficient Action Tokenization for Vision-Language-Action Modelscited by 665
  5. 2022BC-Z: Zero-Shot Task Generalization with Robotic Imitation Learningcited by 523
  6. 2024CogACT: A Foundational Vision-Language-Action Model for Synergizing Cognition and Action in Robotic Manipulationcited by 422
  7. 2024Mobile ALOHA: Learning Bimanual Mobile Manipulation with Low-Cost Whole-Body Teleoperationcited by 391
  8. 2024Robotic Control via Embodied Chain-of-Thought Reasoningcited by 346
  9. 2024TinyVLA: Towards Fast, Data-Efficient Vision-Language-Action Models for Robotic Manipulationcited by 339
  10. 2024RT-H: Action Hierarchies Using Languagecited by 210
  11. 2024PIVOT: Iterative Visual Prompting Elicits Actionable Knowledge for VLMscited by 190
  12. 2023Open-World Object Manipulation using Pre-trained Vision-Language Modelscited by 156
  13. 2020Language-Conditioned Imitation Learning for Robot Manipulation Taskscited by 136
  14. 2025HAMSTER: Hierarchical Action Models For Open-World Robot Manipulationcited by 130
  15. 2024Yell At Your Robot: Improving On-the-Fly from Language Correctionscited by 124
  16. 2025Universal Actions for Enhanced Embodied Foundation Modelscited by 82
  17. 2024LLM3:Large Language Model-based Task and Motion Planning with Motion Failure Reasoningcited by 63
  18. 2024RACER: Rich Language-Guided Failure Recovery Policies for Imitation Learningcited by 49
  19. 2024Closed-Loop Open-Vocabulary Mobile Manipulation with GPT-4Vcited by 40
  20. 2024BUMBLE: Unifying Reasoning and Acting with Vision-Language Models for Building-wide Mobile Manipulationcited by 32
  21. 2023Interactive Robot Learning from Verbal Correctioncited by 29
  22. 2024RLVF: Learning from Verbal Feedback without Overgeneralizationcited by 19
  23. 2024Automating Robot Failure Recovery Using Vision-Language Models With Optimized Promptscited by 16
  24. 2024Open-vocabulary Mobile Manipulation in Unseen Dynamic Environments with 3D Semantic Mapscited by 13
  25. 2022Learning Neuro-symbolic Programs for Language Guided Robot Manipulationcited by 10
  26. 2019Inferring Compact Representations for Efficient Natural Language Understanding of Robot Instructionscited by 9
  27. 2024Robi Butler: Multimodal Remote Interaction with a Household Robot Assistantcited by 6
  28. 2024Training Fast Robot Policies with Slow Foundation Modelscited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 93; 2026: 128.

221 papers.

Sort:

By area:

  1. 2025π0.5: a Vision-Language-Action Model with Open-World Generalizationcited by 1,761
  2. 2026World Action Models are Zero-shot Policiescited by 465
  3. 2025DreamVLA: A Vision-Language-Action Model Dreamed with Comprehensive World Knowledgecited by 213
  4. 2025Ctrl-World: A Controllable Generative World Model for Robot Manipulationcited by 167
  5. 2025Cosmos-Reason1: From Physical Common Sense To Embodied Reasoningcited by 164
  6. 2025ThinkAct: Vision-Language-Action Reasoning via Reinforced Visual Latent Planningcited by 156
  7. 2026π0.7: a Steerable Generalist Robotic Foundation Model with Emergent Capabilitiescited by 138
  8. 2026StarVLA: A Lego-like Codebase for Vision-Language-Action Model Developingcited by 112
  9. 2025OneTwoVLA: A Unified Vision-Language-Action Model with Adaptive Reasoningcited by 107
  10. 2025A Survey on Vision-Language-Action Models: An Action Tokenization Perspectivecited by 96
  11. 2025Galaxea Open-World Dataset and G0 Dual-System VLA Modelcited by 92
  12. 2025Vision-Language-Action Models for Robotics: A Review Towards Real-World Applicationscited by 84
38 more
  1. 2025InternVLA-M1: A Spatially Guided Vision-Language-Action Framework for Generalist Robot Policycited by 84
  2. 2025Vision-Language-Action (VLA) Models: Concepts, Progress, Applications and Challengescited by 74
  3. 2025CogVLA: Cognition-Aligned Vision-Language-Action Model via Instruction-Driven Routing & Sparsificationcited by 74
  4. 2025SafeVLA: Towards Safety Alignment of Vision-Language-Action Model via Constrained Learningcited by 72
  5. 2025InstructVLA: Vision-Language-Action Instruction Tuning from Understanding to Manipulationcited by 72
  6. 2025Large VLM-based Vision-Language-Action Models for Robotic Manipulation: A Surveycited by 69
  7. 2025MemER: Scaling Up Memory for Robot Control via Experience Retrievalcited by 67
  8. 2025RoboMonkey: Scaling Test-Time Sampling and Verification for Vision-Language-Action Modelscited by 61
  9. 2025SP-VLA: A Joint Model Scheduling and Token Pruning Approach for VLA Model Accelerationcited by 58
  10. 2025Hume: Introducing System-2 Thinking in Visual-Language-Action Modelcited by 56
  11. 2025A Survey: Learning Embodied Intelligence from Physical Simulators and World Modelscited by 51
  12. 2026A Survey of Agentic Reasoning for Large Language Models: Towards Recursively Self-Improving and Collective Agentscited by 50
  13. 2025TGRPO :Fine-tuning Vision-Language-Action Model via Trajectory-wise Group Relative Policy Optimizationcited by 49
  14. 2025Pure Vision Language Action (VLA) Models: A Comprehensive Surveycited by 49
  15. 2025Leave No Observation Behind: Real-time Correction for VLA Action Chunkscited by 46
  16. 2025DynaGuide: Steering Diffusion Polices with Active Dynamic Guidancecited by 44
  17. 2025MM-IFEngine: Towards Multimodal Instruction Followingcited by 42
  18. 2025Evo-1: Lightweight Vision-Language-Action Model with Preserved Semantic Alignmentcited by 41
  19. 2026VLAW: Iterative Co-Improvement of Vision-Language-Action Policy and World Modelcited by 41
  20. 2025Vision Language Action Models in Robotic Manipulation: A Systematic Reviewcited by 38
  21. 2025CO-RFT: Efficient Fine-Tuning of Vision-Language-Action Models through Chunked Offline Reinforcement Learningcited by 37
  22. 2025VLA-OS: Structuring and Dissecting Planning Representations and Paradigms in Vision-Language-Action Modelscited by 36
  23. 2026Native Video-Action Pretraining for Generalizable Robot Controlcited by 36
  24. 2025RoboCerebra: A Large-scale Benchmark for Long-horizon Robotic Manipulation Evaluationcited by 34
  25. 2025Thinking vs. Doing: Agents that Reason by Scaling Test-Time Interactioncited by 34
  26. 2025A Survey on Efficient Vision-Language-Action Modelscited by 34
  27. 2025Multimodal Spatial Reasoning in the Large Model Era: A Survey and Benchmarkscited by 33
  28. 2026RoboClaw: An Agentic Framework for Scalable Long-Horizon Robotic Taskscited by 33
  29. 2025Shortcut Learning in Generalist Robot Policies: The Role of Dataset Diversity and Fragmentationcited by 31
  30. 2025VLA-Arena: An Open-Source Framework for Benchmarking Vision-Language-Action Modelscited by 31
  31. 2026When Vision Overrides Language: Evaluating and Mitigating Counterfactual Failures in VLAscited by 30
  32. 2025RoboOS: A Hierarchical Embodied Framework for Cross-Embodiment and Multi-Agent Collaborationcited by 28
  33. 2026Fast-ThinkAct: Efficient Vision-Language-Action Reasoning via Verbalizable Latent Planningcited by 24
  34. 2025Robix: A Unified Model for Robot Interaction, Reasoning and Planningcited by 23
  35. 2025VOTE: Vision-Language-Action Optimization with Trajectory Ensemble Votingcited by 22
  36. 2026A Systematic Study of Data Modalities and Strategies for Co-training Large Behavior Models for Robot Manipulationcited by 22
  37. 2025InSpire: Vision-Language-Action Models with Intrinsic Spatial Reasoningcited by 21
  38. 2025Robot-R1: Reinforcement Learning for Enhanced Embodied Reasoning in Roboticscited by 21

50 most cited of 221 All 221 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026A Brain-inspired Hierarchical Framework for Zero-Shot Robot Task Reasoning and Execution ↗
  2. 2026InstructMove: A Text-Indispensable Benchmark for Instruction-Following Manipulation
  3. 2026Hierarchical Prompting with Dual LLM Modules for Robotic Task and Motion Planning ↗
  4. 2024Talk Through It: End User Directed Manipulation Learning ↗
  5. 2024Embodied Instruction Following in Unknown Environments
  6. 2022Inner Monologue: Embodied Reasoning through Planning with Language Models
  7. 2026Generalizing Manipulation Skills with a Local Coding Agent
  8. 2023ThinkBot: Embodied Instruction Following with Thought Chain Reasoning