mlsanityData to 7 Oct 2026

InstructVLA: Vision-Language-Action Instruction Tuning from Understanding to Manipulation

arXiv 2507.17520 ↗First posted 23 Jul 2025cs.RO cs.CVcited by 71

On the map: LLM reasoning → Spatial and embodied reasoning

also in 1 more
Abstract (arXiv)

To operate effectively in the real world, robots should integrate multimodal reasoning with precise action generation. However, existing vision-language-action (VLA) models often sacrifice one for the other, narrow their abilities to task-specific manipulation data, and suffer catastrophic forgetting of pre-trained vision-language capabilities. To bridge this gap, we introduce InstructVLA, an end-to-end VLA model that preserves the flexible reasoning of large vision-language models (VLMs) while delivering leading manipulation performance with the help of embodied reasoning. InstructVLA introduces a novel training paradigm, Vision-Language-Action Instruction Tuning (VLA-IT), which employs multimodal training with mixture-of-experts adaptation to jointly optimize embodied reasoning and action generation on both standard VLM corpora and a curated 650K-sample VLA-IT dataset. On in-domain SimplerEnv tasks, InstructVLA achieves 33% improvement over SpatialVLA. To evaluate generalization, we introduce SimplerEnv-Instruct, an 80-task benchmark requiring closed-loop control and high-level instruction understanding, where it outperforms a fine-tuned OpenVLA by 96% and an action expert aided by GPT-4o by 29%. Additionally, InstructVLA surpasses baseline VLMs on multimodal tasks and exhibits inference-time scaling by leveraging textual reasoning to boost manipulation performance in both simulated and real-world settings. These results demonstrate InstructVLA's potential for bridging intuitive and steerable human-robot interaction with efficient policy learning.

Key facts. Cited by 71 papers in the map (data through 7 Oct 2026). Placed in Spatial and embodied reasoning, one of 17 directions in LLM reasoning. Builds on 76 of its 92 references that are in the map. First posted on arXiv 23 Jul 2025.

Builds on

76 of 92 references.

Closest ancestors · 4

Same direction, newest first

  1. 2025CoT-VLA: Visual Chain-of-Thought Reasoning for Vision-Language-Action Models27 Mar
  2. 2025Fine-Tuning Vision-Language-Action Models: Optimizing Speed and Success27 Feb
  3. 2024Emma-X: An Embodied Multimodal Action Model with Grounded Chain of Thought and Look-ahead Spatial Reasoning16 Dec
  4. 2024Robotic Control via Embodied Chain-of-Thought Reasoning11 Jul

Other references · 72

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  5. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  6. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  7. 2023Visual Instruction Tuningcited by 7,792
  8. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  9. 2022Scalable Diffusion Models with Transformerscited by 5,931
  10. 2022Flow Matching for Generative Modelingcited by 5,426
  11. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  12. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
38 more
  1. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
  2. 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,777
  3. 2024π0: A Vision-Language-Action Flow Model for General Robot Controlcited by 2,297
  4. 2017FiLM: Visual Reasoning with a General Conditioning Layercited by 2,286
  5. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,168
  6. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,105
  7. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  8. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,007
  9. 2019Towards VQA Models That Can Readcited by 1,696
  10. 2025π0.5: a Vision-Language-Action Model with Open-World Generalizationcited by 1,653
  11. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,624
  12. 2023Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardwarecited by 1,490
  13. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,449
  14. 2024Octo: An Open-Source Generalist Robot Policycited by 1,427
  15. 2023LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learningcited by 1,338
  16. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,288
  17. 2025GR00T N1: An Open Foundation Model for Generalist Humanoid Robotscited by 1,167
  18. 2024Are We on the Right Way for Evaluating Large Vision-Language Models?cited by 893
  19. 2025Emerging Properties in Unified Multimodal Pretrainingcited by 855
  20. 2024DROID: A Large-Scale In-The-Wild Robot Manipulation Datasetcited by 801
  21. 2016A Diagram Is Worth A Dozen Imagescited by 793
  22. 2024PaliGemma: A versatile 3B VLM for transfercited by 679
  23. 2025FAST: Efficient Action Tokenization for Vision-Language-Action Modelscited by 643
  24. 2022Mixture-of-Experts with Expert Choice Routingcited by 611
  25. 2023HallusionBench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Modelscited by 572
  26. 2025SpatialVLA: Exploring Spatial Representations for Visual-Language-Action Modelcited by 502
  27. 2024Evaluating Real-World Robot Manipulation Policies in Simulationcited by 484
  28. 2024Transfusion: Predict the Next Token and Diffuse Images with One Multi-Modal Modelcited by 468
  29. 2023OCRBench: On the Hidden Mystery of OCR in Large Multimodal Modelscited by 466
  30. 2024VLMEvalKit: An Open-Source Toolkit for Evaluating Large Multi-Modality Modelscited by 443
  31. 2021InfographicVQAcited by 430
  32. 2024CogACT: A Foundational Vision-Language-Action Model for Synergizing Cognition and Action in Robotic Manipulationcited by 411
  33. 2024Prismatic VLMs: Investigating the Design Space of Visually-Conditioned Language Modelscited by 336
  34. 2024Latent Action Pretraining from Videoscited by 302
  35. 2021Bridge Data: Boosting Generalization of Robotic Skills with Cross-Domain Datasetscited by 248
  36. 2025Transfer between Modalities with MetaQueriescited by 231
  37. 2025Hi Robot: Open-Ended Instruction Following with Hierarchical Vision-Language-Action Modelscited by 216
  38. 2024RT-H: Action Hierarchies Using Languagecited by 204

50 most cited of 72 All 72 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 20; 2026: 51.

71 papers.

Sort:

By area:

  1. 2026Causal World Modeling for Robot Controlcited by 289
  2. 2025MemoryVLA: Perceptual-Cognitive Memory in Vision-Language-Action Models for Robotic Manipulationcited by 183
  3. 2026StarVLA: A Lego-like Codebase for Vision-Language-Action Model Developingcited by 103
  4. 2025Discrete Diffusion VLA: Bringing Discrete Diffusion to Action Decoding in Vision-Language-Action Policiescited by 90
  5. 2025F1: A Vision-Language-Action Model Bridging Understanding and Generation to Actionscited by 85
  6. 2025InternVLA-M1: A Spatially Guided Vision-Language-Action Framework for Generalist Robot Policycited by 83
  7. 2026InternVLA-A1: Unifying Understanding, Generation and Action for Robotic Manipulationcited by 48
  8. 2025Pure Vision Language Action (VLA) Models: A Comprehensive Surveycited by 46
  9. 2025InternData-A1: Pioneering High-Fidelity Synthetic Data for Pre-training Generalist Policycited by 46
  10. 2025A Survey on Efficient Vision-Language-Action Modelscited by 34
  11. 2026Native Video-Action Pretraining for Generalizable Robot Controlcited by 30
  12. 2026ReMem-VLA: Empowering Vision-Language-Action Model with Memory via Dual-Level Recurrent Queriescited by 28
38 more
  1. 2026StarVLA-α: Reducing Complexity in Vision-Language-Action Systemscited by 27
  2. 2026JEPA-VLA: Video Predictive Embedding is Needed for VLA Modelscited by 22
  3. 2026A Systematic Study of Data Modalities and Strategies for Co-training Large Behavior Models for Robot Manipulationcited by 20
  4. 2025A Survey on Agentic Multimodal Large Language Modelscited by 19
  5. 2025Seeing to Act, Prompting to Specify: A Bayesian Factorization of Vision Language Action Policycited by 17
  6. 2025Contrastive Representation Regularization for Vision-Language-Action Modelscited by 16
  7. 2025PixelVLA: Advancing Pixel-level Understanding in Vision-Language-Action Modelcited by 16
  8. 2025XR-1: Towards Versatile Vision-Language-Action Models via Learning Unified Vision-Motion Representationscited by 15
  9. 2026Scaling Verification Can Be More Effective than Scaling Policy Learning for Vision-Language-Action Alignmentcited by 15
  10. 2026TwinBrainVLA: Unleashing the Potential of Generalist VLMs for Embodied Tasks via Asymmetric Mixture-of-Transformerscited by 14
  11. 2026FutureVLA: Joint Visuomotor Prediction for Vision-Language-Action Modelcited by 14
  12. 2025ManualVLA: A Unified VLA Model for Chain-of-Thought Manual Generation and Robotic Manipulationcited by 13
  13. 2025MM-ACT: Learn from Multimodal Parallel Generation to Actcited by 12
  14. 2026RoboInter: A Holistic Intermediate Representation Suite Towards Robotic Manipulationcited by 12
  15. 2025DualVLA: Building a Generalizable Embodied Agent via Partial Decoupling of Reasoning and Actioncited by 11
  16. 2026ST4VLA: Spatially Guided Training for Vision-Language-Action Modelscited by 8
  17. 2026ProcVLM: Learning Procedure-Grounded Progress Rewards for Robotic Manipulationcited by 8
  18. 2025Unified Embodied VLM Reasoning with Robotic Action via Autoregressive Discretized Pre-trainingcited by 7
  19. 2026PoseVLA: Universal Pose Pretraining for Generalizable Vision-Language-Action Policiescited by 7
  20. 2026TTT-VLA: Test-Time Latent Prompt Optimization for Vision-Language-Action Modelscited by 7
  21. 2026PokeVLA: Empowering Pocket-Sized Vision-Language-Action Model with Comprehensive World Knowledge Guidancecited by 6
  22. 2025CollabVLA: Self-Reflective Vision-Language-Action Model Dreaming Together with Humancited by 5
  23. 2026ReViP: Mitigating False Completion in Vision-Language-Action Models with Vision-Proprioception Rebalancecited by 5
  24. 2026TRAP: Hijacking VLA CoT-Reasoning via Adversarial Patchescited by 5
  25. 2026APT: Action Expert Pretraining Improves Instruction Generalization of Vision-Language-Action Policiescited by 5
  26. 2025An Anatomy of Vision-Language-Action Models: From Modules to Milestones and Challengescited by 4
  27. 2026CF-VLA: Efficient Coarse-to-Fine Action Generation for Vision-Language-Action Policiescited by 4
  28. 2026Closed-Loop Action Chunks with Dynamic Corrections for Training-Free Diffusion Policycited by 3
  29. 2026RoVLA: Multi-Consistency Constraints for Robust Vision-Language-Action Modelscited by 3
  30. 2026VLA-Hijack: A Transferable Patch Attack against Vision-Language-Action Models via Visual Proprioception Hijackingcited by 3
  31. 2026LabVLA: Grounding Vision-Language-Action Models in Scientific Laboratoriescited by 3
  32. 2025IntentionVLA: Generalizable and Efficient Embodied Intention Reasoning for Human-Robot Interactioncited by 2
  33. 2026QuoVLA: Quotient Space for Vision-Language-Action Modelscited by 2
  34. 2026VLA-Pro: Cross-Task Procedural Memory Transfer for Vision-Language-Action Modelscited by 2
  35. 2026Continuous Reasoning for Vision-Language-Actioncited by 2
  36. 2026Unleashing More Actions via Action Compositional Training for VLA Modelscited by 2
  37. 2025Evaluating Multimodal Large Language Models with Daily Composite Tasks in Home Environmentscited by 1
  38. 2026From Perception to Planning: Evolving Ego-Centric Task-Oriented Spatiotemporal Reasoning via Curriculum Learningcited by 1

50 most cited of 71 All 71 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Large VLM-based Vision-Language-Action Models for Robotic Manipulation: A Survey
  2. 2025Vision Language Action Models in Robotic Manipulation: A Systematic Review
  3. 2025Efficient Vision-Language-Action Models for Embodied Manipulation: A Systematic Survey
  4. 2026VLA-Thinker: Boosting Vision-Language-Action Models through Thinking-with-Image Reasoning
  5. 2025ReFineVLA: Reasoning-Aware Teacher-Guided Transfer Fine-Tuning
  6. 2026ReFineVLA: Multimodal Reasoning-Aware Generalist Robotic Policies via Teacher-Guided Fine-Tuning
  7. 2024TinyVLA: Towards Fast, Data-Efficient Vision-Language-Action Models for Robotic Manipulation
  8. 2025Igniting VLMs toward the Embodied Space