mlsanityData to 14 Oct 2026

Enhancing Generalization in Vision-Language-Action Models by Preserving Pretrained Representations

arXiv 2509.11417 ↗First posted 14 Sep 2025cs.RO cs.AI cs.CVcited by 25

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

also in 1 more
Abstract (arXiv)

Vision-language-action (VLA) models finetuned from vision-language models (VLMs) hold the promise of leveraging rich pretrained representations to build generalist robots across diverse tasks and environments. However, direct fine-tuning on robot data often disrupts these representations and limits generalization. We present a framework that better preserves pretrained features while adapting them for robot manipulation. Our approach introduces three components: (i) a dual-encoder design with one frozen vision encoder to retain pretrained features and another trainable for task adaptation, (ii) a string-based action tokenizer that casts continuous actions into character sequences aligned with the model's pretraining domain, and (iii) a co-training strategy that combines robot demonstrations with vision-language datasets emphasizing spatial reasoning and affordances. Evaluations in simulation and on real robots show that our method improves robustness to visual perturbations, generalization to novel instructions and environments, and overall task success compared to baselines.

Key facts. Cited by 25 papers in the map (data through 14 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 52 of its 63 references that are in the map. First posted on arXiv 14 Sep 2025.

Builds on

52 of 63 references.

Closest ancestors · 6

Same direction, newest first

  1. 2025DexVLA: Vision-Language Model with Plug-In Diffusion Expert for General Robot Control9 Feb
  2. 2024LLARVA: Vision-Action Instruction Tuning Enhances Robot Learning17 Jun
  3. 2024RoboPoint: A Vision-Language Model for Spatial Affordance Prediction for Robotics15 Jun
  4. 2024SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilities22 Jan
  5. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control28 Jul
  6. 2022Robotic Skill Acquisition via Instruction Augmentation with Vision-Language Models21 Nov

Other references · 46

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  4. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,974
  5. 2023Visual Instruction Tuningcited by 7,869
  6. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  7. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  8. 2025Qwen2.5-VL Technical Reportcited by 5,587
  9. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  10. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  11. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
  12. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
34 more
  1. 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,887
  2. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,719
  3. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  4. 2024π0: A Vision-Language-Action Flow Model for General Robot Controlcited by 2,407
  5. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,061
  6. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,917
  7. 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,834
  8. 2024Mixtral of Expertscited by 1,823
  9. 2025π0.5: a Vision-Language-Action Model with Open-World Generalizationcited by 1,761
  10. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,729
  11. 2019Towards VQA Models That Can Readcited by 1,718
  12. 2024Octo: An Open-Source Generalist Robot Policycited by 1,484
  13. 2025GR00T N1: An Open Foundation Model for Generalist Humanoid Robotscited by 1,225
  14. 2024MiniCPM-V: A GPT-4V Level MLLM on Your Phonecited by 1,079
  15. 2024Gemma: Open Models Based on Gemini Research and Technologycited by 916
  16. 2024PaliGemma: A versatile 3B VLM for transfercited by 696
  17. 2023VILA: On Pre-training for Visual Language Modelscited by 692
  18. 2023BridgeData V2: A Dataset for Robot Learning at Scalecited by 657
  19. 2023Textbooks Are All You Needcited by 569
  20. 2022Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulationcited by 551
  21. 2024Evaluating Real-World Robot Manipulation Policies in Simulationcited by 495
  22. 2022VIMA: General Robot Manipulation with Multimodal Promptscited by 407
  23. 2024Prismatic VLMs: Investigating the Design Space of Visually-Conditioned Language Modelscited by 339
  24. 2024GR-2: A Generative Video-Language-Action Model with Web-Scale Knowledge for Robot Manipulationcited by 296
  25. 2024CogVLM2: Visual Language Models for Image and Video Understandingcited by 217
  26. 2025HybridVLA: Collaborative Diffusion and Autoregression in a Unified Vision-Language-Action Modelcited by 181
  27. 2023Scaling Robot Learning with Semantically Imagined Experiencecited by 177
  28. 2025Magma: A Foundation Model for Multimodal AI Agentscited by 139
  29. 2024RVT-2: Learning Precise Manipulation from Few Demonstrationscited by 137
  30. 2023RT-Trajectory: Robotic Task Generalization via Hindsight Trajectory Sketchescited by 133
  31. 2025HAMSTER: Hierarchical Action Models For Open-World Robot Manipulationcited by 130
  32. 2025OpenHelix: A Short Survey, Empirical Analysis, and Open-Source Dual-System VLA Model for Robotic Manipulationcited by 82
  33. 2024Theia: Distilling Diverse Vision Foundation Models for Robot Learningcited by 61
  34. 2021Deep Visual Navigation under Partial Observabilitycited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 5; 2026: 20.

25 papers.

Sort:

By area:

  1. 2026π0.7: a Steerable Generalist Robotic Foundation Model with Emergent Capabilitiescited by 138
  2. 2025Actions as Language: Fine-Tuning VLMs into VLAs Without Catastrophic Forgettingcited by 58
  3. 2026Scaling Verification Can Be More Effective than Scaling Policy Learning for Vision-Language-Action Alignmentcited by 21
  4. 2026LAP: Language-Action Pre-Training Enables Zero-shot Cross-Embodiment Transfercited by 18
  5. 2026Scaling World Model for Hierarchical Manipulation Policiescited by 16
  6. 2026Sparse Autoencoders Reveal Interpretable and Steerable Features in VLA Modelscited by 16
  7. 2025Mechanistic Finetuning of Vision-Language-Action Models via Few-Shot Demonstrationscited by 15
  8. 2025Scaling Cross-Embodiment World Models for Dexterous Manipulationcited by 13
  9. 2026VLA-Trace: Diagnosing Vision-Language-Action Models through Representation and Behavior Tracingcited by 13
  10. 2025MAPS: Preserving Vision-Language Representations via Module-Wise Proximity Scheduling for Better Vision-Language-Action Generalizationcited by 7
  11. 2026Point Tracking Improves World Action Modelscited by 7
  12. 2025How Do VLAs Effectively Inherit from VLMs?cited by 6
13 more
  1. 2026Breaking Lock-In: Preserving Steerability under Low-Data VLA Post-Trainingcited by 6
  2. 2026Generalizable VLA Finetuning via Representation Anchoring and Language-Action Alignmentcited by 6
  3. 2026Decoupling the Declarative from the Procedural in Vision-Language-Action Modelscited by 1
  4. 2026Flatness Preserves Instruction Following in Vision-Language-Action Modelscited by 1
  5. 2026UniDexTok: A Unified Dexterous Hand Tokenizer from Real Datacited by 0
  6. 2026CLAP: Direct VLM-to-VLA Adaptation via Language-Action Groundingcited by 0
  7. 2026Visual Cue Guided Video Planning for Generalizable Robot Navigationcited by 0
  8. 2026Same Scene, Different Task: Skill Alignment for Compositional Generalization in VLAscited by 0
  9. 2026How (and How Not) to Use Data Augmentation in VLA Post-Trainingcited by 0
  10. 2026Do VLAs Understand and Adapt to the Objects They Handle, or Simply Replay Learned Behaviors?cited by 0
  11. 2026EgoLAP: Learning from Egocentric Human Data through Language-Action Reasoningcited by 0
  12. 2026When Listening Becomes Easier: Scrubbing Visual Cues for Shortcut-Free VLAscited by 0
  13. 2026RARA: Robust Policy Fine-Tuning through Retrieval-Anchored Representation Alignment for Robotic Manipulationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026PriorVLA: Prior-Preserving Adaptation for Vision-Language-Action Models
  2. 2025From Intention to Execution: Probing the Generalization Boundaries of Vision-Language-Action Models
  3. 2024CogACT: A Foundational Vision-Language-Action Model for Synergizing Cognition and Action in Robotic Manipulation
  4. 2026Beyond Data Scaling: Representation-Centric Continued Pre-training for Vision-Language-Action Models
  5. 2026LA4VLA: Learning to Act without Seeing via Language-Action Pretraining
  6. 2026V-Link: Recovering Lost Visual Representations in Action DiT for Vision-Language-Action Models
  7. 2026Semantic Anchoring for Robotic Action Representations
  8. 2023Programmatically Grounded, Compositionally Generalizable Robotic Manipulation