mlsanityData to 14 Oct 2026

3D-VLA: A 3D Vision-Language-Action Generative World Model

arXiv 2403.09631 ↗First posted 14 Mar 2024cs.CV cs.AI cs.CLcited by 328

On the map: World models → Embodied and robot world models

also in 1 more
Abstract (arXiv)

Recent vision-language-action (VLA) models rely on 2D inputs, lacking integration with the broader realm of the 3D physical world. Furthermore, they perform action prediction by learning a direct mapping from perception to action, neglecting the vast dynamics of the world and the relations between actions and dynamics. In contrast, human beings are endowed with world models that depict imagination about future scenarios to plan actions accordingly. To this end, we propose 3D-VLA by introducing a new family of embodied foundation models that seamlessly link 3D perception, reasoning, and action through a generative world model. Specifically, 3D-VLA is built on top of a 3D-based large language model (LLM), and a set of interaction tokens is introduced to engage with the embodied environment. Furthermore, to inject generation abilities into the model, we train a series of embodied diffusion models and align them into the LLM for predicting the goal images and point clouds. To train our 3D-VLA, we curate a large-scale 3D embodied instruction dataset by extracting vast 3D-related information from existing robotics datasets. Our experiments on held-in datasets demonstrate that 3D-VLA significantly improves the reasoning, multimodal generation, and planning capabilities in embodied environments, showcasing its potential in real-world applications.

Key facts. Cited by 328 papers in the map (data through 14 Oct 2026). Placed in Embodied and robot world models, one of 11 directions in World models. Builds on 38 of its 48 references that are in the map. First posted on arXiv 14 Mar 2024.

Builds on

38 of 48 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023Zero-Shot Robotic Manipulation with Pretrained Image-Editing Diffusion Models16 Oct
  2. 2022BC-Z: Zero-Shot Task Generalization with Robotic Imitation Learning4 Feb

Other references · 36

  1. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  2. 2023Visual Instruction Tuningcited by 7,869
  3. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  4. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  5. 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,117
  6. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 3,080
  7. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  8. 2020RAFT: Recurrent All-Pairs Field Transforms for Optical Flowcited by 2,415
  9. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,071
  10. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,061
  11. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,089
  12. 2023Open X-Embodiment: Robotic Learning Datasets and RT-X Modelscited by 862
24 more
  1. 2018Scaling Egocentric Vision: The EPIC-KITCHENS Datasetcited by 795
  2. 2023Large Language Modelscited by 780
  3. 2022Point-E: A System for Generating 3D Point Clouds from Complex Promptscited by 699
  4. 2019RLBench: The Robot Learning Benchmark & Learning Environmentcited by 689
  5. 2023ZoeDepth: Zero-shot Transfer by Combining Relative and Metric Depthcited by 657
  6. 2023BridgeData V2: A Dataset for Robot Learning at Scalecited by 657
  7. 2023NExT-GPT: Any-to-Any Multimodal LLMcited by 615
  8. 2021CALVIN: A Benchmark for Language-Conditioned Policy Learning for Long-Horizon Robot Manipulation Taskscited by 584
  9. 2023Language Is Not All You Need: Aligning Perception with Language Modelscited by 543
  10. 2022Unified-IO: A Unified Model for Vision, Language, and Multi-Modal Taskscited by 421
  11. 2023An Embodied Generalist Agent in 3D Worldcited by 365
  12. 2022HOI4D: A 4D Egocentric Dataset for Category-Level Human-Object Interactioncited by 327
  13. 2023DreamLLM: Synergistic Multimodal Comprehension and Creationcited by 297
  14. 2022Interactive Language: Talking to Robots in Real Timecited by 218
  15. 2020Language Conditioned Imitation Learning over Unstructured Datacited by 209
  16. 2023RoboVQA: Multimodal Long-Horizon Reasoning for Roboticscited by 120
  17. 2022Grounding Language with Visual Affordances over Unstructured Datacited by 101
  18. 2023On Bringing Robots Homecited by 101
  19. 2023MUTEX: Learning Unified Policies from Multimodal Task Specificationscited by 70
  20. 2023PlayFusion: Skill Acquisition via Diffusion from Language-Annotated Playcited by 50
  21. 2023Finetuning Offline World Models in the Real Worldcited by 41
  22. 2024MultiPLY: A Multisensory Object-Centric Embodied Large Language Model in 3D Worldcited by 39
  23. 2023CoVLM: Composing Visual Entities and Relationships in Large Language Models Via Communicative Decodingcited by 20
  24. 2021Playing with Food: Learning Food Item Representations through Interactive Explorationcited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 32; 2025: 141; 2026: 155.

328 papers.

Sort:

By area:

  1. 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,887
  2. 2025π0.5: a Vision-Language-Action Model with Open-World Generalizationcited by 1,761
  3. 2025GR00T N1: An Open Foundation Model for Generalist Humanoid Robotscited by 1,225
  4. 2025Fine-Tuning Vision-Language-Action Models: Optimizing Speed and Successcited by 883
  5. 2025FAST: Efficient Action Tokenization for Vision-Language-Action Modelscited by 665
  6. 2025SpatialVLA: Exploring Spatial Representations for Visual-Language-Action Modelcited by 519
  7. 2025CoT-VLA: Visual Chain-of-Thought Reasoning for Vision-Language-Action Modelscited by 451
  8. 2025WorldVLA: Towards Autoregressive Action World Modelcited by 367
  9. 2026Causal World Modeling for Robot Controlcited by 321
  10. 2025X-VLA: Soft-Prompted Transformer as Scalable Cross-Embodiment Vision-Language-Action Modelcited by 303
  11. 2024A Survey on Vision-Language-Action Models for Embodied AIcited by 279
  12. 2025MoGe-2: Accurate Monocular Geometry with Metric Scale and Sharp Detailscited by 261
38 more
  1. 2024ShowUI: One Vision-Language-Action Model for GUI Visual Agentcited by 232
  2. 2025Real-Time Execution of Action Chunking Flow Policiescited by 216
  3. 2025DreamVLA: A Vision-Language-Action Model Dreamed with Comprehensive World Knowledgecited by 213
  4. 2025Imagine while Reasoning in Space: Multimodal Visualization-of-Thoughtcited by 206
  5. 2025DexVLA: Vision-Language Model with Plug-In Diffusion Expert for General Robot Controlcited by 195
  6. 2024Aligning Cyber Space with Physical World: A Comprehensive Survey on Embodied AIcited by 179
  7. 2024Understanding World or Predicting Future? A Comprehensive Survey of World Modelscited by 166
  8. 2025Knowledge Insulating Vision-Language-Action Models: Train Fast, Run Fast, Generalize Bettercited by 146
  9. 2026π0.7: a Steerable Generalist Robotic Foundation Model with Emergent Capabilitiescited by 138
  10. 2025Unified Vision-Language-Action Modelcited by 134
  11. 2025Spatial Forcing: Implicit Spatial Representation Alignment for Vision-language-action Modelcited by 134
  12. 2025GraspVLA: a Grasping Foundation Model Pre-trained on Billion-scale Synthetic Action Datacited by 123
  13. 2026VLA-JEPA: Enhancing Vision-Language-Action Model with Latent World Modelcited by 118
  14. 2025AdaWorld: Learning Adaptable World Models with Latent Actionscited by 114
  15. 2025FLARE: Robot Learning with Implicit World Modelingcited by 113
  16. 2024MeshXL: Neural Coordinate Field for Generative 3D Foundation Modelscited by 103
  17. 2025UP-VLA: A Unified Understanding and Prediction Model for Embodied Agentcited by 102
  18. 2025TesserAct: Learning 4D Embodied World Modelscited by 94
  19. 2024What Matters in Building Vision-Language-Action Models for Generalist Robotscited by 89
  20. 2024Pandora: Towards General World Model with Natural Language Actions and Video Statescited by 86
  21. 2025Vision-Language-Action Models for Robotics: A Review Towards Real-World Applicationscited by 84
  22. 2024OAKINK2: A Dataset of Bimanual Hands-Object Manipulation in Complex Task Completioncited by 82
  23. 2025ReconVLA: Reconstructive Vision-Language-Action Model as Effective Robot Perceivercited by 81
  24. 2025BridgeVLA: Input-Output Alignment for Efficient 3D Manipulation Learning with Vision-Language Modelscited by 80
  25. 2025SAFE: Multitask Failure Detection for Vision-Language-Action Modelscited by 80
  26. 2025Vision-Language-Action (VLA) Models: Concepts, Progress, Applications and Challengescited by 74
  27. 2025RynnVLA-002: A Unified Vision-Language-Action and World Modelcited by 74
  28. 2024Towards Synergistic, Generalized, and Efficient Dual-System for Robotic Manipulationcited by 73
  29. 2025A Survey on Vision-Language-Action Models for Autonomous Drivingcited by 72
  30. 2026Unified 4D World Action Modeling from Video Priors with Asynchronous Denoisingcited by 71
  31. 2025VQ-VLA: Improving Vision-Language-Action Models via Scaling Vector-Quantized Action Tokenizerscited by 70
  32. 2025GeoVLA: Empowering 3D Representations in Vision-Language-Action Modelscited by 70
  33. 2024Moto: Latent Motion Token as the Bridging Language for Learning Robot Manipulation from Videoscited by 69
  34. 2025Large VLM-based Vision-Language-Action Models for Robotic Manipulation: A Surveycited by 69
  35. 2025RoboArena: Distributed Real-World Evaluation of Generalist Robot Policiescited by 64
  36. 2025Training-Time Action Conditioning for Efficient Real-Time Chunkingcited by 64
  37. 2026MEM: Multi-Scale Embodied Memory for Vision Language Action Modelscited by 61
  38. 2025Evo-0: Vision-Language-Action Model with Implicit Spatial Understandingcited by 58

50 most cited of 328 All 328 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 202510 Open Challenges Steering the Future of Vision-Language-Action Models
  2. 2026World-Language-Action Model for Unified World Modeling, Language Reasoning, and Action Synthesis
  3. 2025GigaWorld-0: World Models as Data Engine to Empower Embodied AI
  4. 2025Efficient Vision-Language-Action Models for Embodied Manipulation: A Systematic Survey
  5. 2025A Survey on Efficient Vision-Language-Action Models
  6. 2026StarVLA: A Lego-like Codebase for Vision-Language-Action Model Developing
  7. 2025Vision Language Action Models in Robotic Manipulation: A Systematic Review
  8. 2026Towards VLA-Dreamer: Refining VLA Behavior Using World Models