mlsanityData to 7 Oct 2026

HAMSTER: Hierarchical Action Models For Open-World Robot Manipulation

arXiv 2502.05485 ↗First posted 8 Feb 2025cs.RO cs.AI cs.CVcited by 125

On the map: Vision-language-action models → Generalist VLA models and benchmarks

Abstract (arXiv)

Large foundation models have shown strong open-world generalization to complex problems in vision and language, but similar levels of generalization have yet to be achieved in robotics. One fundamental challenge is the lack of robotic data, which are typically obtained through expensive on-robot operation. A promising remedy is to leverage cheaper, off-domain data such as action-free videos, hand-drawn sketches or simulation data. In this work, we posit that hierarchical vision-language-action (VLA) models can be more effective in utilizing off-domain data than standard monolithic VLA models that directly finetune vision-language models (VLMs) to predict actions. In particular, we study a class of hierarchical VLA models, where the high-level VLM is finetuned to produce a coarse 2D path indicating the desired robot end-effector trajectory given an RGB image and a task description. The intermediate 2D path prediction is then served as guidance to the low-level, 3D-aware control policy capable of precise manipulation. Doing so alleviates the high-level VLM from fine-grained action prediction, while reducing the low-level policy's burden on complex task-level reasoning. We show that, with the hierarchical design, the high-level VLM can transfer across significant domain gaps between the off-domain finetuning data and real-robot testing scenarios, including differences on embodiments, dynamics, visual appearances and task semantics, etc. In the real-robot experiments, we observe an average of 20% improvement in success rate across seven different axes of generalization over OpenVLA, representing a 50% relative gain. Visual results, code, and dataset are provided at: https://hamster-robot.github.io/

Key facts. Cited by 125 papers in the map (data through 7 Oct 2026). Placed in Generalist VLA models and benchmarks, one of 13 directions in Vision-language-action models. Builds on 71 of its 84 references that are in the map. First posted on arXiv 8 Feb 2025.

Builds on

71 of 84 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024π0: A Vision-Language-Action Flow Model for General Robot Control31 Oct
  2. 2024OpenVLA: An Open-Source Vision-Language-Action Model13 Jun
  3. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control28 Jul

Other references · 68

  1. 2023Visual Instruction Tuningcited by 7,792
  2. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  3. 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusioncited by 2,730
  4. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  5. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  6. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,105
  7. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  8. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  9. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,007
  10. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  11. 2019Towards VQA Models That Can Readcited by 1,696
  12. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
38 more
  1. 2023Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardwarecited by 1,490
  2. 2024Octo: An Open-Source Generalist Robot Policycited by 1,427
  3. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,221
  4. 2022Code as Policies: Language Model Programs for Embodied Controlcited by 1,172
  5. 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,057
  6. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,029
  7. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehensioncited by 929
  8. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 852
  9. 2024DROID: A Large-Scale In-The-Wild Robot Manipulation Datasetcited by 801
  10. 2023VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Modelscited by 733
  11. 2021What Matters in Learning from Offline Human Demonstrations for Robot Manipulationcited by 699
  12. 2023VILA: On Pre-training for Visual Language Modelscited by 686
  13. 2022R3M: A Universal Visual Representation for Robot Manipulationcited by 678
  14. 2019RLBench: The Robot Learning Benchmark & Learning Environmentcited by 669
  15. 2023BridgeData V2: A Dataset for Robot Learning at Scalecited by 634
  16. 2022ProgPrompt: Generating Situated Robot Task Plans using Large Language Modelscited by 598
  17. 2022Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulationcited by 545
  18. 2023Eureka: Human-Level Reward Design via Coding Large Language Modelscited by 542
  19. 2022BC-Z: Zero-Shot Task Generalization with Robotic Imitation Learningcited by 517
  20. 2023CoTracker: It is Better to Track Togethercited by 516
  21. 2020Learning Quadrupedal Locomotion over Challenging Terraincited by 494
  22. 2024Evaluating Real-World Robot Manipulation Policies in Simulationcited by 484
  23. 2022VIMA: General Robot Manipulation with Multimodal Promptscited by 403
  24. 2022VIP: Towards Universal Visual Reward and Representation via Value-Implicit Pre-Trainingcited by 384
  25. 2023Scaling Open-Vocabulary Object Detectioncited by 335
  26. 2023MimicGen: A Data Generation System for Scalable Robot Learning using Human Demonstrationscited by 308
  27. 2023Language to Rewards for Robotic Skill Synthesiscited by 307
  28. 2023TAPIR: Tracking Any Point with per-frame Initialization and temporal Refinementcited by 281
  29. 20243D Diffuser Actor: Policy Diffusion with 3D Scene Representationscited by 278
  30. 2023Reward Design with Language Modelscited by 257
  31. 2024RoboPoint: A Vision-Language Model for Spatial Affordance Prediction for Roboticscited by 242
  32. 2023RVT: Robotic View Transformer for 3D Object Manipulationcited by 224
  33. 2023Text2Motion: From Natural Language Instructions to Feasible Planscited by 223
  34. 2023Multimodal C4: An Open, Billion-scale Corpus of Images Interleaved with Textcited by 208
  35. 2023Any-point Trajectory Modeling for Policy Learningcited by 207
  36. 2023Tracking Everything Everywhere All at Oncecited by 190
  37. 2024PIVOT: Iterative Visual Prompting Elicits Actionable Knowledge for VLMscited by 187
  38. 2022The Unsurprising Effectiveness of Pre-Trained Vision Models for Controlcited by 155

50 most cited of 68 All 68 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 55; 2026: 70.

125 papers.

Sort:

By area:

  1. 2025π0.5: a Vision-Language-Action Model with Open-World Generalizationcited by 1,653
  2. 2026World Action Models are Zero-shot Policiescited by 411
  3. 2025Hi Robot: Open-Ended Instruction Following with Hierarchical Vision-Language-Action Modelscited by 216
  4. 2025MolmoAct: Action Reasoning Models that can Reason in Spacecited by 189
  5. 2025Cosmos-Reason1: From Physical Common Sense To Embodied Reasoningcited by 159
  6. 2025ThinkAct: Vision-Language-Action Reasoning via Reinforced Visual Latent Planningcited by 152
  7. 2025OneTwoVLA: A Unified Vision-Language-Action Model with Adaptive Reasoningcited by 105
  8. 2025InternVLA-M1: A Spatially Guided Vision-Language-Action Framework for Generalist Robot Policycited by 83
  9. 2025Vision-Language-Action Models for Robotics: A Review Towards Real-World Applicationscited by 82
  10. 2025Vision-Language-Action (VLA) Models: Concepts, Progress, Applications and Challengescited by 72
  11. 2025InstructVLA: Vision-Language-Action Instruction Tuning from Understanding to Manipulationcited by 71
  12. 2025CogVLA: Cognition-Aligned Vision-Language-Action Model via Instruction-Driven Routing & Sparsificationcited by 70
38 more
  1. 2025Large VLM-based Vision-Language-Action Models for Robotic Manipulation: A Surveycited by 66
  2. 2025MemER: Scaling Up Memory for Robot Control via Experience Retrievalcited by 64
  3. 2026Robometer: Scaling General-Purpose Robotic Reward Models via Trajectory Comparisonscited by 51
  4. 2025VLA-0: Building State-of-the-Art VLAs with Zero Modificationcited by 49
  5. 2026InternVLA-A1: Unifying Understanding, Generation and Action for Robotic Manipulationcited by 48
  6. 2025Pure Vision Language Action (VLA) Models: A Comprehensive Surveycited by 46
  7. 2026MolmoAct2: Action Reasoning Models for Real-world Deploymentcited by 45
  8. 2025LeVERB: Humanoid Whole-Body Control with Latent Vision-Language Instructioncited by 40
  9. 2025Vision Language Action Models in Robotic Manipulation: A Systematic Reviewcited by 38
  10. 2025Embodied-R1: Reinforced Embodied Reasoning for General Robotic Manipulationcited by 36
  11. 2025From Seeing to Doing: Bridging Reasoning and Decision for Robotic Manipulationcited by 34
  12. 2025VLA-OS: Structuring and Dissecting Planning Representations and Paradigms in Vision-Language-Action Modelscited by 34
  13. 2025A Survey on Efficient Vision-Language-Action Modelscited by 34
  14. 2026RoboClaw: An Agentic Framework for Scalable Long-Horizon Robotic Taskscited by 30
  15. 2025Learning Long-Context Diffusion Policies via Past-Token Predictioncited by 28
  16. 2025LoHoVLA: A Unified Vision-Language-Action Model for Long-Horizon Embodied Taskscited by 25
  17. 2025VOTE: Vision-Language-Action Optimization with Trajectory Ensemble Votingcited by 22
  18. 2025Enhancing Generalization in Vision-Language-Action Models by Preserving Pretrained Representationscited by 21
  19. 2026Fast-ThinkAct: Efficient Vision-Language-Action Reasoning via Verbalizable Latent Planningcited by 20
  20. 2026A Systematic Study of Data Modalities and Strategies for Co-training Large Behavior Models for Robot Manipulationcited by 20
  21. 2026Steerable Vision-Language-Action Policies for Embodied Reasoning and Hierarchical Controlcited by 19
  22. 2025Parallels Between VLA Model Post-Training and Human Motor Learning: Progress, Challenges, and Trendscited by 17
  23. 2026VP-VLA: Visual Prompting as an Interface for Vision-Language-Action Modelscited by 17
  24. 2026G0.5: One Autoregressive Stream for Robot Reasoning and Actioncited by 17
  25. 2025TraceGen: World Modeling in 3D Trace Space Enables Learning from Cross-Embodiment Videoscited by 13
  26. 2025A Survey on Imitation Learning for Contact-Rich Tasks in Roboticscited by 12
  27. 2025Towards Spatial Trace with Reasoning in Vision-Language Models for Roboticscited by 12
  28. 2026RoboInter: A Holistic Intermediate Representation Suite Towards Robotic Manipulationcited by 12
  29. 2025PEEK: Guiding and Minimal Image Representations for Zero-Shot Generalization of Robot Manipulation Policiescited by 11
  30. 2025Point-It-Out: Benchmarking Embodied Reasoning for Vision Language Models in Multi-Stage Visual Groundingcited by 10
  31. 2025Clutter-Robust Vision-Language-Action Models through Object-Centric and Geometry Groundingcited by 10
  32. 2025Continual Learning for Generative AI: From LLMs to MLLMs and Beyondcited by 9
  33. 2026Long-Horizon Manipulation via Trace-Conditioned VLA Planningcited by 9
  34. 2026What Matters in Orchestrating Robot Policies: A Systematic Study of Hierarchical VLA Agentscited by 9
  35. 2026ST4VLA: Spatially Guided Training for Vision-Language-Action Modelscited by 8
  36. 2025Efficient Evaluation of Multi-Task Robot Policies With Active Experiment Selectioncited by 7
  37. 2025Pixel Motion Diffusion is What We Need for Robot Controlcited by 7
  38. 2025ExpReS-VLA: Specializing Vision-Language-Action Models Through Experience Replay and Retrievalcited by 7

50 most cited of 125 All 125 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Scaling World Model for Hierarchical Manipulation Policies
  2. 2026H-VLA: Hierarchical Vision-Language-Action Model with Key-Action Reasoning and Motion Planning in a Unified Action Space
  3. 2026GeneralVLA: Generalizable Vision-Language-Action Models with Knowledge-Guided Trajectory Planning
  4. 2024CogACT: A Foundational Vision-Language-Action Model for Synergizing Cognition and Action in Robotic Manipulation
  5. 2025Efficient Vision-Language-Action Models for Embodied Manipulation: A Systematic Survey
  6. 2024Benchmarking Vision, Language, & Action Models on Robotic Learning Tasks
  7. 2026Vision-Language-Action Models: Experimental Insights from a Real-World UR5 Platform
  8. 2024What Matters in Building Vision-Language-Action Models for Generalist Robots