mlsanityData to 7 Oct 2026

What Matters in Building Vision-Language-Action Models for Generalist Robots

arXiv 2412.14058 ↗First posted 18 Dec 2024cs.RO cs.CVcited by 88

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

also in 1 more
Abstract (arXiv)

To utilize Foundation Vision Language Models (VLMs) for robotic tasks and motion planning, the community has proposed different methods for injecting action components into VLMs and building the Vision-Language-Action models (VLAs). In this work, we disclose the key factors that significantly influence the performance of VLA on robot manipulation problems and focus on answering three essential design choices: which backbone to select, how to formulate the VLA architectures, and when to add cross-embodiment data. The obtained results convince us firmly to explain why we prefer VLA and develop a new family of VLAs, RoboVLMs, which require very few manual designs and achieve a new state-of-the-art performance in three simulation tasks and real-world experiments. Through our extensive experiments, which include over 8 VLM backbones, 4 policy architectures, and over 600 distinct designed experiments, we provide a detailed guidebook for the future design of VLAs. In addition to the study, the highly flexible RoboVLMs framework, which supports easy integrations of new VLMs and free combinations of various design choices, is made public to facilitate future research. We open-source all details, including codes, models, datasets, and toolkits, along with detailed training and evaluation recipes at: robovlms.github.io.

Key facts. Cited by 88 papers in the map (data through 7 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 42 of its 55 references that are in the map. First posted on arXiv 18 Dec 2024.

Builds on

42 of 55 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024DeeR-VLA: Dynamic Inference of Multimodal Large Language Models for Efficient Robot Execution4 Nov
  2. 2024Robotic Control via Embodied Chain-of-Thought Reasoning11 Jul
  3. 2024RoboUniView: Visual-Language Model with Unified View Representation for Robotic Manipulation27 Jun
  4. 2023Vision-Language Foundation Models as Effective Robot Imitators2 Nov
  5. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control28 Jul

Other references · 37

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2023Visual Instruction Tuningcited by 7,792
  4. 2022Flow Matching for Generative Modelingcited by 5,426
  5. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,861
  6. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  7. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  8. 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,777
  9. 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusioncited by 2,730
  10. 2014Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modelingcited by 2,341
  11. 2024π0: A Vision-Language-Action Flow Model for General Robot Controlcited by 2,297
  12. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,007
25 more
  1. 2023Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardwarecited by 1,490
  2. 2024Octo: An Open-Source Generalist Robot Policycited by 1,427
  3. 2021Perceiver: General Perception with Iterative Attentioncited by 1,089
  4. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,075
  5. 2022A Generalist Agentcited by 802
  6. 2024PaliGemma: A versatile 3B VLM for transfercited by 679
  7. 2022R3M: A Universal Visual Representation for Robot Manipulationcited by 678
  8. 2023The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)cited by 677
  9. 2023BridgeData V2: A Dataset for Robot Learning at Scalecited by 634
  10. 2021CALVIN: A Benchmark for Language-Conditioned Policy Learning for Long-Horizon Robot Manipulation Taskscited by 564
  11. 2022BC-Z: Zero-Shot Task Generalization with Robotic Imitation Learningcited by 517
  12. 2024Evaluating Real-World Robot Manipulation Policies in Simulationcited by 484
  13. 2022VIMA: General Robot Manipulation with Multimodal Promptscited by 403
  14. 2020Sim-to-Real Transfer in Deep Reinforcement Learning for Robotics: a Surveycited by 388
  15. 2021Attention Bottlenecks for Multimodal Fusioncited by 361
  16. 2023Unleashing Large-Scale Video Generative Pre-training for Visual Robot Manipulationcited by 353
  17. 20243D-VLA: A 3D Vision-Language-Action Generative World Modelcited by 320
  18. 2024Latent Action Pretraining from Videoscited by 302
  19. 2024GR-2: A Generative Video-Language-Action Model with Web-Scale Knowledge for Robot Manipulationcited by 290
  20. 20243D Diffuser Actor: Policy Diffusion with 3D Scene Representationscited by 278
  21. 2022Real-World Robot Learning with Masked Visual Pre-trainingcited by 241
  22. 2015Variable Rate Image Compression with Recurrent Neural Networkscited by 168
  23. 2024Reconciling Reality through Simulation: A Real-to-Sim-to-Real Approach for Robust Manipulationcited by 135
  24. 2021VLM: Task-agnostic Video-Language Model Pre-training for Video Understandingcited by 102
  25. 2024GR-MG: Leveraging Partially Annotated Data via Multi-Modal Goal-Conditioned Policycited by 39

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 60; 2026: 28.

88 papers.

Sort:

By area:

  1. 2025SpatialVLA: Exploring Spatial Representations for Visual-Language-Action Modelcited by 502
  2. 2025UniVLA: Learning to Act Anywhere with Task-centric Latent Actionscited by 386
  3. 2025MemoryVLA: Perceptual-Cognitive Memory in Vision-Language-Action Models for Robotic Manipulationcited by 183
  4. 2025HybridVLA: Collaborative Diffusion and Autoregression in a Unified Vision-Language-Action Modelcited by 176
  5. 2025VLA-Adapter: An Effective Paradigm for Tiny-Scale Vision-Language-Action Modelcited by 140
  6. 2025ConRFT: A Reinforced Fine-tuning Method for VLA Models via Consistency Policycited by 139
  7. 2025Unified Vision-Language-Action Modelcited by 128
  8. 2025Spatial Forcing: Implicit Spatial Representation Alignment for Vision-language-action Modelcited by 125
  9. 2025GR-3 Technical Reportcited by 110
  10. 2026VLA-JEPA: Enhancing Vision-Language-Action Model with Latent World Modelcited by 104
  11. 2025F1: A Vision-Language-Action Model Bridging Understanding and Generation to Actionscited by 85
  12. 2025Dita: Scaling Diffusion Transformer for Generalist Vision-Language-Action Policycited by 73
38 more
  1. 2025villa-X: Enhancing Latent Action Modeling in Vision-Language-Action Modelscited by 58
  2. 2025Hume: Introducing System-2 Thinking in Visual-Language-Action Modelcited by 56
  3. 2025FLOWER: Democratizing Generalist Robot Policies with Efficient Vision-Language-Action Flow Policiescited by 56
  4. 2025FlowVLA: Visual Chain of Thought-based Motion Reasoning for Vision-Language-Action Modelscited by 50
  5. 2025Long-VLA: Unleashing Long-Horizon Capability of Vision Language Action Model for Robot Manipulationcited by 48
  6. 2025GR-RL: Going Dexterous and Precise for Long-Horizon Robotic Manipulationcited by 41
  7. 2025SpecPrune-VLA: Accelerating Vision-Language-Action Models via Action-Aware Self-Speculative Pruningcited by 39
  8. 2026VLM4VLA: Revisiting Vision-Language-Models in Vision-Language-Action Modelscited by 38
  9. 2025From Seeing to Doing: Bridging Reasoning and Decision for Robotic Manipulationcited by 34
  10. 2025CO-RFT: Efficient Fine-Tuning of Vision-Language-Action Models through Chunked Offline Reinforcement Learningcited by 34
  11. 2025ContextVLA: Vision-Language-Action Model with Amortized Multi-Frame Contextcited by 34
  12. 2025LLaDA-VLA: Vision Language Diffusion Action Modelscited by 30
  13. 2025Learning Long-Context Diffusion Policies via Past-Token Predictioncited by 28
  14. 2025JARVIS-VLA: Post-Training Large-Scale Vision Language Models to Play Visual Games with Keyboards and Mousecited by 27
  15. 2025NORA-1.5: A Vision-Language-Action Model Trained using World Model- and Action-based Preference Rewardscited by 26
  16. 2025LoHoVLA: A Unified Vision-Language-Action Model for Long-Horizon Embodied Taskscited by 25
  17. 2025From Spatial to Actions: Grounding Vision-Language-Action Model in Spatial Foundation Priorscited by 25
  18. 2025CronusVLA: Towards Efficient and Robust Manipulation via Multi-Frame Vision-Language-Action Modelingcited by 24
  19. 2025STARE-VLA: Progressive Stage-Aware Reinforcement for Fine-Tuning Vision-Language-Action Modelscited by 24
  20. 2025FreezeVLA: Action-Freezing Attacks against Vision-Language-Action Modelscited by 19
  21. 2025Steering Vision-Language-Action Models as Anti-Exploration: A Test-Time Scaling Approachcited by 18
  22. 2025Dream-VL & Dream-VLA: Open Vision-Language and Vision-Language-Action Models with Diffusion Language Model Backbonecited by 18
  23. 2025OmniVGGT: Omni-Modality Driven Visual Geometry Grounded Transformercited by 17
  24. 2025Seeing to Act, Prompting to Specify: A Bayesian Factorization of Vision Language Action Policycited by 17
  25. 2026Agent Banana: High-Fidelity Image Editing with Agentic Thinking and Toolingcited by 17
  26. 2026Scaling World Model for Hierarchical Manipulation Policiescited by 15
  27. 2026TwinBrainVLA: Unleashing the Potential of Generalist VLMs for Embodied Tasks via Asymmetric Mixture-of-Transformerscited by 14
  28. 2025Unveiling the Potential of Vision-Language-Action Models with Open-Ended Multimodal Instructionscited by 12
  29. 2025MoA-VR: A Mixture-of-Agents System Towards All-in-One Video Restorationcited by 12
  30. 2026RoboInter: A Holistic Intermediate Representation Suite Towards Robotic Manipulationcited by 12
  31. 2025Spatial Traces: Enhancing VLA Models with Spatial-Temporal Understandingcited by 11
  32. 2025DualVLA: Building a Generalizable Embodied Agent via Partial Decoupling of Reasoning and Actioncited by 11
  33. 2026LaMP: Learning Vision-Language-Action Policy with 3D Scene Flow as Latent Motion Priorcited by 11
  34. 2025PhysBrain: Human Egocentric Data as a Bridge from Vision Language Models to Physical Intelligencecited by 10
  35. 2025X-IL: Exploring the Design Space of Imitation Learning Policiescited by 9
  36. 2025Adversarial Data Collection: Human-Collaborative Perturbations for Efficient and Robust Robotic Imitation Learningcited by 9
  37. 2025RoboChemist: Long-Horizon and Safety-Compliant Robotic Chemical Experimentationcited by 9
  38. 2025OpenHA: A Series of Open-Source Hierarchical Agentic Models in Minecraftcited by 9

50 most cited of 88 All 88 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Vision Language Action Models in Robotic Manipulation: A Systematic Review
  2. 2025Vision-Language-Action Models for Robotics: A Review Towards Real-World Applications
  3. 2025VLA-0: Building State-of-the-Art VLAs with Zero Modification
  4. 2024Benchmarking Vision, Language, & Action Models on Robotic Learning Tasks
  5. 2025An Anatomy of Vision-Language-Action Models: From Modules to Milestones and Challenges
  6. 2025Pure Vision Language Action (VLA) Models: A Comprehensive Survey
  7. 2025From Intention to Execution: Probing the Generalization Boundaries of Vision-Language-Action Models
  8. 2025Large VLM-based Vision-Language-Action Models for Robotic Manipulation: A Survey