# VLM4VLA: Revisiting Vision-Language-Models in Vision-Language-Action Models

arXiv: https://arxiv.org/abs/2601.03309 · HTML: https://mlsanity.com/paper/2601.03309/

## Key facts

- Cited by 39 papers in the map (data through 14 Oct 2026).
- Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 46 of its 49 references that are in the map.
- First posted on arXiv 6 Jan 2026.

## Placed in

- Multimodal LLMs / vision-language models / [Embodied and 3D vision-language models](https://mlsanity.com/mllm/embodied-and-3d-vision-language-models/)
- Vision-language-action models / [VLA architectures and generalization](https://mlsanity.com/vla/vla-architectures-and-generalization/)

## Builds on (most cited first, 30 of 46)

- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Qwen2.5-VL Technical Report](https://mlsanity.com/paper/2502.13923/) (2025)
- [RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control](https://mlsanity.com/paper/2307.15818/) (2023)
- [OpenVLA: An Open-Source Vision-Language-Action Model](https://mlsanity.com/paper/2406.09246/) (2024)
- [InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning](https://mlsanity.com/paper/2305.06500/) (2023)
- [Qwen3-VL Technical Report](https://mlsanity.com/paper/2511.21631/) (2025)
- [π0: A Vision-Language-Action Flow Model for General Robot Control](https://mlsanity.com/paper/2410.24164/) (2024)
- [π0.5: a Vision-Language-Action Model with Open-World Generalization](https://mlsanity.com/paper/2504.16054/) (2025)
- [Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware](https://mlsanity.com/paper/2304.13705/) (2023)
- [InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiency](https://mlsanity.com/paper/2508.18265/) (2025)
- [LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learning](https://mlsanity.com/paper/2306.03310/) (2023)
- [Kosmos-2: Grounding Multimodal Large Language Models to the World](https://mlsanity.com/paper/2306.14824/) (2023)
- [Emerging Properties in Unified Multimodal Pretraining](https://mlsanity.com/paper/2505.14683/) (2025)
- [SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilities](https://mlsanity.com/paper/2401.12168/) (2024)
- [PaliGemma: A versatile 3B VLM for transfer](https://mlsanity.com/paper/2407.07726/) (2024)
- [FAST: Efficient Action Tokenization for Vision-Language-Action Models](https://mlsanity.com/paper/2501.09747/) (2025)
- [BridgeData V2: A Dataset for Robot Learning at Scale](https://mlsanity.com/paper/2308.12952/) (2023)
- [Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spaces](https://mlsanity.com/paper/2412.14171/) (2024)
- [CALVIN: A Benchmark for Language-Conditioned Policy Learning for Long-Horizon Robot Manipulation Tasks](https://mlsanity.com/paper/2112.03227/) (2021)
- [Evaluating Real-World Robot Manipulation Policies in Simulation](https://mlsanity.com/paper/2405.05941/) (2024)
- [BLIP3-o: A Family of Fully Open Unified Multimodal Models-Architecture, Training and Dataset](https://mlsanity.com/paper/2505.09568/) (2025)
- [Vision-Language Foundation Models as Effective Robot Imitators](https://mlsanity.com/paper/2311.01378/) (2023)
- [TraceVLA: Visual Trace Prompting Enhances Spatial-Temporal Awareness for Generalist Robotic Policies](https://mlsanity.com/paper/2412.10345/) (2024)
- [SayPlan: Grounding Large Language Models using 3D Scene Graphs for Scalable Robot Task Planning](https://mlsanity.com/paper/2307.06135/) (2023)
- [Video Prediction Policy: A Generalist Robot Policy with Predictive Visual Representations](https://mlsanity.com/paper/2412.14803/) (2024)
- [RoboPoint: A Vision-Language Model for Spatial Affordance Prediction for Robotics](https://mlsanity.com/paper/2406.10721/) (2024)
- [PaliGemma 2: A Family of Versatile VLMs for Transfer](https://mlsanity.com/paper/2412.03555/) (2024)
- [DexVLA: Vision-Language Model with Plug-In Diffusion Expert for General Robot Control](https://mlsanity.com/paper/2502.05855/) (2025)
- [MemoryVLA: Perceptual-Cognitive Memory in Vision-Language-Action Models for Robotic Manipulation](https://mlsanity.com/paper/2508.19236/) (2025)
- [SpaceR: Reinforcing MLLMs in Video Spatial Reasoning](https://mlsanity.com/paper/2504.01805/) (2025)

## Built on by (most cited first, 30 of 39)

- [Qwen-VLA: Unifying Vision-Language-Action Modeling across Tasks, Environments, and Robot Embodiments](https://mlsanity.com/paper/2605.30280/) (2026)
- [ReMem-VLA: Empowering Vision-Language-Action Model with Memory via Dual-Level Recurrent Queries](https://mlsanity.com/paper/2603.12942/) (2026)
- [Not All Features Are Created Equal: A Mechanistic Study of Vision-Language-Action Models](https://mlsanity.com/paper/2603.19233/) (2026)
- [WAM4D: Fast 4D World Action Model via Spatial Register Tokens](https://mlsanity.com/paper/2606.14048/) (2026)
- [TwinBrainVLA: Unleashing the Potential of Generalist VLMs for Embodied Tasks via Asymmetric Mixture-of-Transformers](https://mlsanity.com/paper/2601.14133/) (2026)
- [VLANeXt: Recipes for Building Strong VLA Models](https://mlsanity.com/paper/2602.18532/) (2026)
- [Goal2Skill: Long-Horizon Manipulation with Adaptive Planning and Reflection](https://mlsanity.com/paper/2604.13942/) (2026)
- [Rethinking the Practicality of Vision-language-action Model: A Comprehensive Benchmark and An Improved Baseline](https://mlsanity.com/paper/2602.22663/) (2026)
- [Hide-and-Seek in Trajectories: Discovering Failure Signals for VLA Runtime Monitoring](https://mlsanity.com/paper/2605.30834/) (2026)
- [Rethinking Muon Beyond Pretraining: Spectral Failures and High-Pass Remedies for VLA and RLVR](https://mlsanity.com/paper/2605.19282/) (2026)
- [PoseVLA: Universal Pose Pretraining for Generalizable Vision-Language-Action Policies](https://mlsanity.com/paper/2602.19710/) (2026)
- [Revisiting Embodied Chain-of-Thought for Generalizable Robot Manipulation](https://mlsanity.com/paper/2606.03784/) (2026)
- [Embodied-R1.5: Evolving Physical Intelligence via Embodied Foundation Models](https://mlsanity.com/paper/2606.11324/) (2026)
- [Veo-Act: Enhancing VLA Policies with Frontier Video Models](https://mlsanity.com/paper/2604.04502/) (2026)
- [UAM: A Dual-Stream Perspective on Forgetting in VLA Training](https://mlsanity.com/paper/2605.15735/) (2026)
- [OmniStream: Mastering Perception, Reconstruction and Action in Continuous Streams](https://mlsanity.com/paper/2603.12265/) (2026)
- [Look Before Acting: Enhancing Vision Foundation Representations for Vision-Language-Action Models](https://mlsanity.com/paper/2603.15618/) (2026)
- [Escaping the Diversity Trap in Robotic Manipulation via Anchor-Centric Adaptation](https://mlsanity.com/paper/2605.07381/) (2026)
- [MultihopSpatial: Multi-hop Compositional Spatial Reasoning Benchmark for Vision-Language Model](https://mlsanity.com/paper/2603.18892/) (2026)
- [RoboAlign: Learning Test-Time Reasoning for Language-Action Alignment in Vision-Language-Action Models](https://mlsanity.com/paper/2603.21341/) (2026)
- [PhysMem: Scaling Test-Time Memory for Embodied Physical Reasoning](https://mlsanity.com/paper/2602.20323/) (2026)
- [Unmasking the Illusion of Embodied Reasoning in Vision-Language-Action Models](https://mlsanity.com/paper/2604.18000/) (2026)
- [EmbodiedMidtrain: Bridging the Gap between Vision-Language Models and Vision-Language-Action Models via Mid-training](https://mlsanity.com/paper/2604.20012/) (2026)
- [VLA-GSE: Boosting Parameter-Efficient Fine-Tuning in VLA with Generalized and Specialized Experts](https://mlsanity.com/paper/2605.06175/) (2026)
- [QuoVLA: Quotient Space for Vision-Language-Action Models](https://mlsanity.com/paper/2605.24890/) (2026)
- [ELAN4D: Embodiment-Centric 4D Supervision for Vision-Language-Action Models via Plug-and-Play Adaptation](https://mlsanity.com/paper/2605.30484/) (2026)
- [Bimanual Robot Manipulation via Multi-Agent In-Context Learning](https://mlsanity.com/paper/2604.20348/) (2026)
- [Action with Visual Primitives](https://mlsanity.com/paper/2605.22183/) (2026)
- [3DThinkVLA: Endowing Vision-Language-Action Models with Latent 3D Priors via 3D-Thinking-Guided Co-training](https://mlsanity.com/paper/2606.04436/) (2026)
- [SPARC: Reliable Spatial Annotations from Robot Demonstrations at Scale](https://mlsanity.com/paper/2606.13497/) (2026)

## Abstract (arXiv)

Vision-Language-Action (VLA) models, which integrate pretrained large Vision-Language Models (VLM) into their policy backbone, are gaining significant attention for their promising generalization capabilities. This paper revisits a fundamental yet seldom systematically studied question: how VLM choice and competence translate to downstream VLA policies performance? We introduce VLM4VLA, a minimal adaptation pipeline that converts general-purpose VLMs into VLA policies using only a small set of new learnable parameters for fair and efficient comparison. Despite its simplicity, VLM4VLA proves surprisingly competitive with more sophisticated network designs. Through extensive empirical studies on various downstream tasks across three benchmarks, we find that while VLM initialization offers a consistent benefit over training from scratch, a VLM's general capabilities are poor predictors of its downstream task performance. This challenges common assumptions, indicating that standard VLM competence is necessary but insufficient for effective embodied control. We further investigate the impact of specific embodied capabilities by fine-tuning VLMs on seven auxiliary embodied tasks (e.g., embodied QA, visual pointing, depth estimation). Contrary to intuition, improving a VLM's performance on specific embodied skills does not guarantee better downstream control performance. Finally, modality-level ablations identify the visual module in VLM, rather than the language component, as the primary performance bottleneck. We demonstrate that injecting control-relevant supervision into the vision encoder of the VLM yields consistent gains, even when the encoder remains frozen during downstream fine-tuning. This isolates a persistent domain gap between current VLM pretraining objectives and the requirements of embodied action-planning.