EmbodiedMidtrain: Bridging the Gap between Vision-Language Models and Vision-Language-Action Models via Mid-training
On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models
also in 1 more
Abstract (arXiv)
Vision-Language-Action Models (VLAs) inherit their visual and linguistic capabilities from Vision-Language Models (VLMs), yet most VLAs are built from off-the-shelf VLMs that are not adapted to the embodied domain, limiting their downstream performance. In this work, we propose EmbodiedMidtrain to bridge the gap between VLMs and VLAs. We first characterize the data distribution gap between them, showing that VLA data occupy compact regions that are largely separated from the broader VLM distribution, while the degree of alignment varies substantially both across and within VLM data sources. Then, we build a mid-training data engine that leverages a lightweight learnable proximity estimator to select the most VLA-aligned candidates from a large VLM pool, and mid-trains the VLM on this curated mixture before downstream VLA fine-tuning. Experiments on three robot manipulation benchmarks show that mid-training consistently improves performance across different VLM backbones, achieving results competitive with expert VLAs and off-the-shelf VLMs trained with larger model scale and training budgets. Further analysis reveals that mid-training provides a stronger initialization for VLA fine-tuning, with gains emerging from the earliest steps and widening throughout training. Moreover, the data engine captures both dataset-level and sample-level alignment signals, favoring spatial reasoning over text-centric tasks while preserving the diversity of the VLM data. We will release all code, data and models for future research.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 45 of its 49 references that are in the map. First posted on arXiv 21 Apr 2026.
Builds on
45 of 49 references.
Closest ancestors · 11
Same direction, newest first
- 2026VLM4VLA: Revisiting Vision-Language-Models in Vision-Language-Action Models6 Jan
- 2025Vlaser: Vision-Language-Action Model with Synergistic Embodied Reasoning13 Oct
- 2025Contrastive Representation Regularization for Vision-Language-Action Models2 Oct
- 2025RoboRefer: Towards Spatial Referring with Reasoning in Vision-Language Models for Robotics4 Jun
- 2025Robo2VLM: Visual Question Answering from Large-Scale In-the-Wild Robot Manipulation Datasets21 May
- 2025RoboBrain: A Unified Brain Model for Robotic Manipulation from Abstract to Concrete28 Feb
- 2025Fine-Tuning Vision-Language-Action Models: Optimizing Speed and Success27 Feb
- 2024Is A Picture Worth A Thousand Words? Delving Into Spatial Reasoning for Vision Language Models21 Jun
- 2024SpatialBot: Precise Spatial Understanding with Vision Language Models19 Jun
- 2024RoboPoint: A Vision-Language Model for Spatial Affordance Prediction for Robotics15 Jun
- 2024EmbSpatial-Bench: Benchmarking Spatial Understanding for Embodied Tasks with Large Vision-Language Models9 Jun
Other references · 34
- 2023Visual Instruction Tuningcited by 7,792
- 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
- 2022Generative Adversarial Networkscited by 5,603
- 2025Qwen2.5-VL Technical Reportcited by 5,418
- 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
- 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
- 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
- 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,777
- 2025Qwen3-VL Technical Reportcited by 2,534
- 2024π0: A Vision-Language-Action Flow Model for General Robot Controlcited by 2,297
- 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
- 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
22 more
- 2025π0.5: a Vision-Language-Action Model with Open-World Generalizationcited by 1,653
- 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,573
- 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
- 2024LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Modelscited by 1,442
- 2025InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiencycited by 1,419
- 2020Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hyperspherecited by 1,393
- 2023LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learningcited by 1,338
- 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
- 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,075
- 2024MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategiescited by 694
- 2024PaliGemma: A versatile 3B VLM for transfercited by 679
- 2023BridgeData V2: A Dataset for Robot Learning at Scalecited by 634
- 2021CALVIN: A Benchmark for Language-Conditioned Policy Learning for Long-Horizon Robot Manipulation Taskscited by 564
- 2018From Recognition to Cognition: Visual Commonsense Reasoningcited by 553
- 2024Evaluating Real-World Robot Manipulation Policies in Simulationcited by 484
- 2023Data Selection for Language Models via Importance Resamplingcited by 322
- 2024PaliGemma 2: A Family of Versatile VLMs for Transfercited by 204
- 20243DSRBench: A Comprehensive 3D Spatial Reasoning Benchmarkcited by 133
- 2025OctoThinker: Mid-training Incentivizes Reinforcement Learning Scalingcited by 110
- 2025VisuLogic: A Benchmark for Evaluating Visual Reasoning in Multi-modal Large Language Modelscited by 102
- 2025Eagle 2: Building Post-Training Data Strategies from Scratch for Frontier Vision-Language Modelscited by 69
- 2025Shortcut Learning in Generalist Robot Policies: The Role of Dataset Diversity and Fragmentationcited by 31
Built on by
Citing papers per year: 2026: 2.
2 papers.
Similar, not cited
Close in wording, no citation either way
- 2026VLAFlow: A Unified Training Framework for Vision-Language-Action Models via Co-training and Future Latent Alignment
- 2026Two Bridges, One Pathway: From VLMs to Generalizable VLAs with Embodied Trajectory-Coupled Data
- 2025Efficient Vision-Language-Action Models for Embodied Manipulation: A Systematic Survey
- 2026Vision-Language-Action in Robotics: A Survey of Datasets, Benchmarks, and Data Engines
- 2025A Survey on Efficient Vision-Language-Action Models
- 2025Large VLM-based Vision-Language-Action Models for Robotic Manipulation: A Survey
- 2026Rethinking Visual-Language-Action Model Scaling: Alignment, Mixture, and Regularization
- 2025IA-VLA: Input Augmentation for Vision-Language-Action models in settings with semantically complex tasks