mlsanityData to 7 Oct 2026

EmbodiedMidtrain: Bridging the Gap between Vision-Language Models and Vision-Language-Action Models via Mid-training

arXiv 2604.20012 ↗First posted 21 Apr 2026cs.CV cs.AI cs.CLcited by 2

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

also in 1 more
Abstract (arXiv)

Vision-Language-Action Models (VLAs) inherit their visual and linguistic capabilities from Vision-Language Models (VLMs), yet most VLAs are built from off-the-shelf VLMs that are not adapted to the embodied domain, limiting their downstream performance. In this work, we propose EmbodiedMidtrain to bridge the gap between VLMs and VLAs. We first characterize the data distribution gap between them, showing that VLA data occupy compact regions that are largely separated from the broader VLM distribution, while the degree of alignment varies substantially both across and within VLM data sources. Then, we build a mid-training data engine that leverages a lightweight learnable proximity estimator to select the most VLA-aligned candidates from a large VLM pool, and mid-trains the VLM on this curated mixture before downstream VLA fine-tuning. Experiments on three robot manipulation benchmarks show that mid-training consistently improves performance across different VLM backbones, achieving results competitive with expert VLAs and off-the-shelf VLMs trained with larger model scale and training budgets. Further analysis reveals that mid-training provides a stronger initialization for VLA fine-tuning, with gains emerging from the earliest steps and widening throughout training. Moreover, the data engine captures both dataset-level and sample-level alignment signals, favoring spatial reasoning over text-centric tasks while preserving the diversity of the VLM data. We will release all code, data and models for future research.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 45 of its 49 references that are in the map. First posted on arXiv 21 Apr 2026.

Builds on

45 of 49 references.

Closest ancestors · 11

Same direction, newest first

  1. 2026VLM4VLA: Revisiting Vision-Language-Models in Vision-Language-Action Models6 Jan
  2. 2025Vlaser: Vision-Language-Action Model with Synergistic Embodied Reasoning13 Oct
  3. 2025Contrastive Representation Regularization for Vision-Language-Action Models2 Oct
  4. 2025RoboRefer: Towards Spatial Referring with Reasoning in Vision-Language Models for Robotics4 Jun
  5. 2025Robo2VLM: Visual Question Answering from Large-Scale In-the-Wild Robot Manipulation Datasets21 May
  6. 2025RoboBrain: A Unified Brain Model for Robotic Manipulation from Abstract to Concrete28 Feb
  7. 2025Fine-Tuning Vision-Language-Action Models: Optimizing Speed and Success27 Feb
  8. 2024Is A Picture Worth A Thousand Words? Delving Into Spatial Reasoning for Vision Language Models21 Jun
  9. 2024SpatialBot: Precise Spatial Understanding with Vision Language Models19 Jun
  10. 2024RoboPoint: A Vision-Language Model for Spatial Affordance Prediction for Robotics15 Jun
  11. 2024EmbSpatial-Bench: Benchmarking Spatial Understanding for Embodied Tasks with Large Vision-Language Models9 Jun

Other references · 34

  1. 2023Visual Instruction Tuningcited by 7,792
  2. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  3. 2022Generative Adversarial Networkscited by 5,603
  4. 2025Qwen2.5-VL Technical Reportcited by 5,418
  5. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  6. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  7. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  8. 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,777
  9. 2025Qwen3-VL Technical Reportcited by 2,534
  10. 2024π0: A Vision-Language-Action Flow Model for General Robot Controlcited by 2,297
  11. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
  12. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
22 more
  1. 2025π0.5: a Vision-Language-Action Model with Open-World Generalizationcited by 1,653
  2. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,573
  3. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
  4. 2024LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Modelscited by 1,442
  5. 2025InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiencycited by 1,419
  6. 2020Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hyperspherecited by 1,393
  7. 2023LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learningcited by 1,338
  8. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
  9. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,075
  10. 2024MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategiescited by 694
  11. 2024PaliGemma: A versatile 3B VLM for transfercited by 679
  12. 2023BridgeData V2: A Dataset for Robot Learning at Scalecited by 634
  13. 2021CALVIN: A Benchmark for Language-Conditioned Policy Learning for Long-Horizon Robot Manipulation Taskscited by 564
  14. 2018From Recognition to Cognition: Visual Commonsense Reasoningcited by 553
  15. 2024Evaluating Real-World Robot Manipulation Policies in Simulationcited by 484
  16. 2023Data Selection for Language Models via Importance Resamplingcited by 322
  17. 2024PaliGemma 2: A Family of Versatile VLMs for Transfercited by 204
  18. 20243DSRBench: A Comprehensive 3D Spatial Reasoning Benchmarkcited by 133
  19. 2025OctoThinker: Mid-training Incentivizes Reinforcement Learning Scalingcited by 110
  20. 2025VisuLogic: A Benchmark for Evaluating Visual Reasoning in Multi-modal Large Language Modelscited by 102
  21. 2025Eagle 2: Building Post-Training Data Strategies from Scratch for Frontier Vision-Language Modelscited by 69
  22. 2025Shortcut Learning in Generalist Robot Policies: The Role of Dataset Diversity and Fragmentationcited by 31

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 2.

2 papers.

Similar, not cited

Close in wording, no citation either way

  1. 2026VLAFlow: A Unified Training Framework for Vision-Language-Action Models via Co-training and Future Latent Alignment
  2. 2026Two Bridges, One Pathway: From VLMs to Generalizable VLAs with Embodied Trajectory-Coupled Data
  3. 2025Efficient Vision-Language-Action Models for Embodied Manipulation: A Systematic Survey
  4. 2026Vision-Language-Action in Robotics: A Survey of Datasets, Benchmarks, and Data Engines
  5. 2025A Survey on Efficient Vision-Language-Action Models
  6. 2025Large VLM-based Vision-Language-Action Models for Robotic Manipulation: A Survey
  7. 2026Rethinking Visual-Language-Action Model Scaling: Alignment, Mixture, and Regularization
  8. 2025IA-VLA: Input Augmentation for Vision-Language-Action models in settings with semantically complex tasks