mlsanityData to 7 Oct 2026

Knowledge Insulating Vision-Language-Action Models: Train Fast, Run Fast, Generalize Better

arXiv 2505.23705 ↗First posted 29 May 2025cs.LG cs.ROcited by 142

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

also in 1 more
Abstract (arXiv)

Vision-language-action (VLA) models provide a powerful approach to training control policies for physical systems, such as robots, by combining end-to-end learning with transfer of semantic knowledge from web-scale vision-language model (VLM) training. However, the constraints of real-time control are often at odds with the design of VLMs: the most powerful VLMs have tens or hundreds of billions of parameters, presenting an obstacle to real-time inference, and operate on discrete tokens rather than the continuous-valued outputs that are required for controlling robots. To address this challenge, recent VLA models have used specialized modules for efficient continuous control, such as action experts or continuous output heads, which typically require adding new untrained parameters to the pretrained VLM backbone. While these modules improve real-time and control capabilities, it remains an open question whether they preserve or degrade the semantic knowledge contained in the pretrained VLM, and what effect they have on the VLA training dynamics. In this paper, we study this question in the context of VLAs that include a continuous diffusion or flow matching action expert, showing that naively including such experts significantly harms both training speed and knowledge transfer. We provide an extensive analysis of various design choices, their impact on performance and knowledge transfer, and propose a technique for insulating the VLM backbone during VLA training that mitigates this issue. Videos are available at https://pi.website/research/knowledge_insulation.

Key facts. Cited by 142 papers in the map (data through 7 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 52 of its 60 references that are in the map. First posted on arXiv 29 May 2025.

Builds on

52 of 60 references.

Closest ancestors · 6

Same direction, newest first

  1. 2025Fine-Tuning Vision-Language-Action Models: Optimizing Speed and Success27 Feb
  2. 2025DexVLA: Vision-Language Model with Plug-In Diffusion Expert for General Robot Control9 Feb
  3. 2024From Multimodal LLMs to Generalist Embodied Agents: Methods and Lessons11 Dec
  4. 2024Robotic Control via Embodied Chain-of-Thought Reasoning11 Jul
  5. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control28 Jul
  6. 2023PaLM-E: An Embodied Multimodal Language Model6 Mar

Other references · 46

  1. 2017Attention Is All You Needcited by 59,932
  2. 2023Visual Instruction Tuningcited by 7,792
  3. 2022Flow Matching for Generative Modelingcited by 5,426
  4. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  5. 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesiscited by 4,094
  6. 2022Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flowcited by 3,550
  7. 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,777
  8. 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusioncited by 2,730
  9. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  10. 2024π0: A Vision-Language-Action Flow Model for General Robot Controlcited by 2,297
  11. 2025π0.5: a Vision-Language-Action Model with Open-World Generalizationcited by 1,653
  12. 2023Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardwarecited by 1,490
34 more
  1. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
  2. 2024Octo: An Open-Source Generalist Robot Policycited by 1,427
  3. 2023LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learningcited by 1,338
  4. 2025GR00T N1: An Open Foundation Model for Generalist Humanoid Robotscited by 1,167
  5. 2024Chameleon: Mixed-Modal Early-Fusion Foundation Modelscited by 904
  6. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMscited by 822
  7. 2024DROID: A Large-Scale In-The-Wild Robot Manipulation Datasetcited by 801
  8. 2024PaliGemma: A versatile 3B VLM for transfercited by 679
  9. 2024RDT-1B: a Diffusion Foundation Model for Bimanual Manipulationcited by 670
  10. 2025FAST: Efficient Action Tokenization for Vision-Language-Action Modelscited by 643
  11. 2023BridgeData V2: A Dataset for Robot Learning at Scalecited by 634
  12. 2024Transfusion: Predict the Next Token and Diffuse Images with One Multi-Modal Modelcited by 468
  13. 2024CogACT: A Foundational Vision-Language-Action Model for Synergizing Cognition and Action in Robotic Manipulationcited by 411
  14. 2025AgiBot World Colosseo: A Large-scale Manipulation Platform for Scalable and Intelligent Embodied Systemscited by 390
  15. 2024TinyVLA: Towards Fast, Data-Efficient Vision-Language-Action Models for Robotic Manipulationcited by 331
  16. 20243D-VLA: A 3D Vision-Language-Action Generative World Modelcited by 320
  17. 2023OBELICS: An Open Web-Scale Filtered Dataset of Interleaved Image-Text Documentscited by 316
  18. 2022Rectified Flow: A Marginal Preserving Approach to Optimal Transportcited by 277
  19. 2019RoboNet: Large-Scale Multi-Robot Learningcited by 268
  20. 2021Bridge Data: Boosting Generalization of Robotic Skills with Cross-Domain Datasetscited by 248
  21. 2021VisualGPT: Data-efficient Adaptation of Pretrained Language Models for Image Captioningcited by 190
  22. 2025HybridVLA: Collaborative Diffusion and Autoregression in a Unified Vision-Language-Action Modelcited by 176
  23. 2023RH20T: A Comprehensive Robotic Dataset for Learning Diverse Skills in One-Shotcited by 173
  24. 2023RoboAgent: Generalization and Efficiency in Robot Manipulation via Semantic Augmentations and Action Chunkingcited by 167
  25. 2024Mixture-of-Transformers: A Sparse and Scalable Architecture for Multi-Modal Foundation Modelscited by 151
  26. 2024ALOHA Unleashed: A Simple Recipe for Robot Dexteritycited by 128
  27. 2024LMFusion: Adapting Pretrained Language Models for Multimodal Generationcited by 120
  28. 2025ChatVLA: Unified Multimodal Understanding and Robot Control with Vision-Language-Action Modelcited by 112
  29. 2023On Bringing Robots Homecited by 101
  30. 2023CapsFusion: Rethinking Image-Text Data at Scalecited by 90
  31. 2024BAKU: An Efficient Transformer for Multi-Task Policy Learningcited by 80
  32. 2025OTTER: A Vision-Language-Action Model with Text-Aware Visual Feature Extractioncited by 54
  33. 2025ChatVLA-2: Vision-Language-Action Model with Open-World Embodied Reasoning from Pretrained Knowledgecited by 49
  34. 2024Efficient Diffusion Transformer Policies with Mixture of Expert Denoisers for Multitask Learningcited by 46

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 42; 2026: 100.

142 papers.

Sort:

By area:

  1. 2025X-VLA: Soft-Prompted Transformer as Scalable Cross-Embodiment Vision-Language-Action Modelcited by 285
  2. 2025π*0.6: a VLA That Learns From Experiencecited by 280
  3. 2025Alpamayo-R1: Bridging Reasoning and Action Prediction for Generalizable Autonomous Driving in the Long Tailcited by 136
  4. 2025mimic-video: Video-Action Models for Generalizable Robot Control Beyond VLAscited by 133
  5. 2026π0.7: a Steerable Generalist Robotic Foundation Model with Emergent Capabilitiescited by 127
  6. 2026StarVLA: A Lego-like Codebase for Vision-Language-Action Model Developingcited by 103
  7. 2025A Survey on Vision-Language-Action Models: An Action Tokenization Perspectivecited by 96
  8. 2025InternVLA-M1: A Spatially Guided Vision-Language-Action Framework for Generalist Robot Policycited by 83
  9. 2025Vision-Language-Action Models for Robotics: A Review Towards Real-World Applicationscited by 82
  10. 2025InstructVLA: Vision-Language-Action Instruction Tuning from Understanding to Manipulationcited by 71
  11. 2025CogVLA: Cognition-Aligned Vision-Language-Action Model via Instruction-Driven Routing & Sparsificationcited by 70
  12. 2025Large VLM-based Vision-Language-Action Models for Robotic Manipulation: A Surveycited by 66
38 more
  1. 2025FLOWER: Democratizing Generalist Robot Policies with Efficient Vision-Language-Action Flow Policiescited by 56
  2. 2026MEM: Multi-Scale Embodied Memory for Vision Language Action Modelscited by 55
  3. 2025VLASH: Real-Time VLAs via Future-State-Aware Asynchronous Inferencecited by 54
  4. 2025VLA-0: Building State-of-the-Art VLAs with Zero Modificationcited by 49
  5. 2026Qwen-RobotManip Technical Report: Alignment Unlocks Scale for Robotic Manipulation Foundation Modelscited by 47
  6. 2025Pure Vision Language Action (VLA) Models: A Comprehensive Surveycited by 46
  7. 2026LaWAM: Latent World Action Models for Efficient Dynamics-Aware Robot Policiescited by 46
  8. 2026MolmoAct2: Action Reasoning Models for Real-world Deploymentcited by 45
  9. 2025GigaBrain-0: A World Model-Powered Vision-Language-Action Modelcited by 43
  10. 2026Xiaomi-Robotics-0: An Open-Sourced Vision-Language-Action Model with Real-Time Executioncited by 42
  11. 2025CEED-VLA: Consistency Vision-Language-Action Model with Early-Exit Decodingcited by 38
  12. 2025Vision Language Action Models in Robotic Manipulation: A Systematic Reviewcited by 38
  13. 2026VLM4VLA: Revisiting Vision-Language-Models in Vision-Language-Action Modelscited by 38
  14. 2025Emergence of Human to Robot Transfer in Vision-Language-Action Modelscited by 36
  15. 2025ContextVLA: Vision-Language-Action Model with Amortized Multi-Frame Contextcited by 34
  16. 2025Don't Blind Your VLA: Aligning Visual Representations for OOD Generalizationcited by 34
  17. 2025EO-1: An Open Unified Embodied Foundation Model for General Robot Controlcited by 30
  18. 2025dVLA: Diffusion Vision-Language-Action Model with Multimodal Chain-of-Thoughtcited by 30
  19. 2025Fast-in-Slow: A Dual-System Foundation Model Unifying Fast Manipulation within Slow Reasoningcited by 28
  20. 2026CLAP: Contrastive Latent Action Pretraining for Learning Vision-Language-Action Models from Human Videoscited by 27
  21. 2026GigaBrain-0.5M*: a VLA That Learns From World Model-Based Reinforcement Learningcited by 23
  22. 2025Vlaser: Vision-Language-Action Model with Synergistic Embodied Reasoningcited by 22
  23. 2026Fast-ThinkAct: Efficient Vision-Language-Action Reasoning via Verbalizable Latent Planningcited by 20
  24. 2026A Systematic Study of Data Modalities and Strategies for Co-training Large Behavior Models for Robot Manipulationcited by 20
  25. 2026Steerable Vision-Language-Action Policies for Embodied Reasoning and Hierarchical Controlcited by 19
  26. 2025Seeing to Act, Prompting to Specify: A Bayesian Factorization of Vision Language Action Policycited by 17
  27. 2026LAP: Language-Action Pre-Training Enables Zero-shot Cross-Embodiment Transfercited by 17
  28. 2026ActionCodec: What Makes for Good Action Tokenizerscited by 17
  29. 2026G0.5: One Autoregressive Stream for Robot Reasoning and Actioncited by 17
  30. 2025Contrastive Representation Regularization for Vision-Language-Action Modelscited by 16
  31. 2026Scaling Verification Can Be More Effective than Scaling Policy Learning for Vision-Language-Action Alignmentcited by 15
  32. 2026FutureVLA: Joint Visuomotor Prediction for Vision-Language-Action Modelcited by 14
  33. 2026Sparse Autoencoders Reveal Interpretable and Steerable Features in VLA Modelscited by 14
  34. 2026VLA-Corrector: Lightweight Detect-and-Correct Inference for Adaptive Action Horizoncited by 14
  35. 2025Unifying Perception and Action: A Hybrid-Modality Pipeline with Implicit Visual Chain-of-Thought for Robotic Action Generationcited by 12
  36. 2026DM0: An Embodied-Native Vision-Language-Action Model towards Physical AIcited by 12
  37. 2026Wall-OSS-0.5 Technical Reportcited by 11
  38. 2026Scaling Behavior Cloning Improves Causal Reasoning: An Open Model for Real-Time Video Game Playingcited by 10

50 most cited of 142 All 142 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Actions as Language: Fine-Tuning VLMs into VLAs Without Catastrophic Forgetting
  2. 2025VLA-Adapter: An Effective Paradigm for Tiny-Scale Vision-Language-Action Model
  3. 2026Does VLA Even Know the Basics? Measuring Commonsense and World Knowledge Retention in Vision-Language-Action Models
  4. 2026Two Bridges, One Pathway: From VLMs to Generalizable VLAs with Embodied Trajectory-Coupled Data
  5. 2026Explicit Language Memory for Long-Horizon Planning in Vision-Language-Action Models
  6. 2026ElegantVLA: Learning When to Think for Efficient Vision-Language-Action Models
  7. 2025Improving Vision-Language-Action Model with Online Reinforcement Learning
  8. 2024What Matters in Building Vision-Language-Action Models for Generalist Robots