mlsanityData to 7 Oct 2026

MaD-Mix: Multi-Modal Data Mixtures via Latent Space Coupling for Vision-Language Model Training

arXiv 2602.07790 ↗First posted 8 Feb 2026cs.LGcited by 0

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Vision-Language Models (VLMs) are typically trained on a diverse set of multi-modal domains, yet current practices rely on costly manual tuning. We propose MaD-Mix, a principled and computationally efficient framework that derives multi-modal data mixtures for VLM training. MaD-Mix formulates data mixing as modality-aware domain alignment maximization and obtains closed-form multi-modal alignment scores from the Fenchel dual through inter-modal coupling variables. MaD-Mix systematically handles domains with missing modalities, allowing for the integration of language-only domains. Empirical evaluations across 0.5B and 7B models demonstrate that MaD-Mix accelerates VLM training across diverse benchmarks. MaD-Mix matches human-tuned data mixtures using 22% fewer training steps in image-text instruction tuning. In complex tri-modal video-image-text scenarios, where manual tuning becomes impractical, MaD-Mix boosts average accuracy over uniform weights, with negligible mixture computation overhead (< 1 GPU-hour), enabling scalable mixture design for modern VLM pipelines.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 109 of its 131 references that are in the map. First posted on arXiv 8 Feb 2026.

Builds on

109 of 131 references.

Closest ancestors · 17

Same direction, newest first

  1. 2025VisualWebInstruct: Scaling up Multimodal Instruction Data through Web Search13 Mar
  2. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling6 Dec
  3. 2024Infinity-MM: Scaling Multimodal Performance with Large-Scale and High-Quality Instruction Data24 Oct
  4. 2024A Survey on Multimodal Benchmarks: In the Era of Large AI Models21 Sep
  5. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution18 Sep
  6. 2024LLaVA-OneVision: Easy Visual Task Transfer6 Aug
  7. 2024LMMs-Eval: Reality Check on the Evaluation of Large Multimodal Models17 Jul
  8. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs24 Jun
  9. 2024A Survey of Multimodal Large Language Model from A Data-centric Perspective26 May
  10. 2024Vision-Flan: Scaling Human-Labeled Tasks in Visual Instruction Tuning18 Feb
  11. 2024Efficient Multimodal Learning from Data-centric Perspective18 Feb
  12. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captions21 Nov
5 more
  1. 2023Improved Baselines with Visual Instruction Tuning5 Oct
  2. 2023MMBench: Is Your Multi-modal Model an All-around Player?12 Jul
  3. 2023A Survey on Multimodal Large Language Models23 Jun
  4. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning11 May
  5. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuning5 May

Other references · 92

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2024The Llama 3 Herd of Modelscited by 14,192
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2025Qwen3 Technical Reportcited by 8,195
  6. 2023Visual Instruction Tuningcited by 7,792
  7. 2020Scaling Laws for Neural Language Modelscited by 6,411
  8. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  9. 2023Qwen Technical Reportcited by 3,273
  10. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  11. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  12. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
38 more
  1. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
  2. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  3. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  4. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,449
  5. 2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,445
  6. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,288
  7. 2019Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalizationcited by 1,232
  8. 2023MVBench: A Comprehensive Multi-modal Video Understanding Benchmarkcited by 1,086
  9. 2017Multimodal Machine Learning: A Survey and Taxonomycited by 1,004
  10. 2024Are We on the Right Way for Evaluating Large Vision-Language Models?cited by 893
  11. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 852
  12. 2016A Diagram Is Worth A Dozen Imagescited by 793
  13. 2021NExT-QA:Next Phase of Question-Answering to Explaining Temporal Actionscited by 735
  14. 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 702
  15. 2023DataComp: In search of the next generation of multimodal datasetscited by 635
  16. 2024MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems?cited by 573
  17. 2019ActivityNet-QA: A Dataset for Understanding Complex Web Videos via Question Answeringcited by 559
  18. 2020The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memescited by 485
  19. 2023OCRBench: On the Hidden Mystery of OCR in Large Multimodal Modelscited by 466
  20. 2020TextCaps: a Dataset for Image Captioning with Reading Comprehensioncited by 398
  21. 2024InternLM2 Technical Reportcited by 391
  22. 2023DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pretrainingcited by 385
  23. 2021Inter-GPS: Interpretable Geometry Problem Solving with Formal Language and Symbolic Reasoningcited by 376
  24. 2022Dynamic Prompt Learning via Policy Gradient for Semi-structured Mathematical Reasoningcited by 347
  25. 2024Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothingcited by 341
  26. 2018DVQA: Understanding Data Visualizations via Question Answeringcited by 338
  27. 2023PMC-VQA: Visual Instruction Tuning for Medical Visual Question Answeringcited by 333
  28. 2024What matters when building vision-language models?cited by 321
  29. 2015Visual7W: Grounded Question Answering in Imagescited by 315
  30. 2017FigureQA: An Annotated Figure Dataset for Visual Reasoningcited by 314
  31. 2023Perception Test: A Diagnostic Benchmark for Multimodal Video Modelscited by 310
  32. 2022Visual Spatial Reasoningcited by 288
  33. 2023LLaVAR: Enhanced Visual Instruction Tuning for Text-Rich Image Understandingcited by 272
  34. 2021IconQA: A New Benchmark for Abstract Diagram Understanding and Visual Language Reasoningcited by 262
  35. 2024A Survey on Data Selection for Language Modelscited by 230
  36. 2019RAVEN: A Dataset for Relational and Analogical Visual rEasoNingcited by 225
  37. 2025Video-MMMU: Evaluating Knowledge Acquisition from Multi-Discipline Professional Videoscited by 225
  38. 2023G-LLaVA: Solving Geometric Problem with Multi-Modal Large Language Modelcited by 208

50 most cited of 92 All 92 (JSON)

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Deciphering Cross-Modal Alignment in Large Vision-Language Models with Modality Integration Rate
  2. 2025Better Reasoning with Less Data: Enhancing VLMs Through Unified Modality Scoring
  3. 2026Cross-Modal Redundancy and the Geometry of Vision-Language Embeddings
  4. 2026DataComp-VLM: Improved Open Datasets for Vision-Language Models
  5. 2025Unified modality separation: A vision-language framework for unsupervised domain adaptation
  6. 2025MoIIE: Mixture of Intra- and Inter-Modality Experts for Large Vision Language Models
  7. 2023TiMix: Text-aware Image Mixing for Effective Vision-Language Pre-training
  8. 2025MergeMix: A Unified Augmentation Paradigm for Visual and Multi-Modal Understanding