mlsanityData to 14 Oct 2026

Data Metabolism: An Efficient Data Design Schema For Vision Language Model

arXiv 2504.12316 ↗First posted 10 Apr 2025cs.CL cs.AI cs.CVcited by 1

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Data curation plays a crucial role in training powerful Visual Language Models (VLMs). In this work, we introduce the concept of Data Metabolism and present our data-centric framework to build VLMs throughout the development lifecycle. Starting from a standard model architecture, we discuss and provide insights into two crucial development steps: data curation and iteration, forming a closed-loop system that continuously improves model performance. We show a detailed codebook on how to process existing massive datasets and build user-specific data flywheel. As a demonstration, we release a VLM, named Capybara-VL, which excels in typical multimodal tasks (e.g. , visual question answering, scientific reasoning, and text-rich tasks). Despite its relatively compact size, Capybara-VL surpasses several open-source models that are up to 10 times larger in size. Moreover, it achieves results that are on par with those of several leading proprietary models, demonstrating its remarkable competitiveness. These results highlight the power of our data-centric framework and the potential of training smaller and more efficient VLMs.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 139 of its 158 references that are in the map. First posted on arXiv 10 Apr 2025.

Builds on

139 of 158 references.

Closest ancestors · 15

Same direction, newest first

  1. 2024LLaVA-CoT: Let Vision Language Models Reason Step-by-Step15 Nov
  2. 2024Unraveling Cross-Modality Knowledge Conflicts in Large Vision-Language Models4 Oct
  3. 2024Building and better understanding vision-language models: insights and future directions22 Aug
  4. 2024MAVIS: Mathematical Visual Instruction Tuning with an Automatic Data Engine11 Jul
  5. 2024WildVision: Evaluating Vision-Language Models in the Wild with Human Preferences16 Jun
  6. 2024From Pixels to Prose: A Large Dataset of Dense Image Captions14 Jun
  7. 2024What matters when building vision-language models?3 May
  8. 2024Are We on the Right Way for Evaluating Large Vision-Language Models?29 Mar
  9. 2024MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems?21 Mar
  10. 2024Multimodal ArXiv: A Dataset for Improving Scientific Comprehension of Large Vision-Language Models1 Mar
  11. 2024Measuring Multimodal Mathematical Reasoning with MATH-Vision Dataset22 Feb
  12. 2024ALLaVA: Harnessing GPT4V-Synthesized Data for Lite Vision-Language Models18 Feb
3 more
  1. 2023CapsFusion: Rethinking Image-Text Data at Scale31 Oct
  2. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond24 Aug
  3. 2022Flamingo: a Visual Language Model for Few-Shot Learning29 Apr

Other references · 124

  1. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  2. 2023Visual Instruction Tuningcited by 7,869
  3. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  4. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  5. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  6. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
  7. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  8. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,719
  9. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,565
  10. 2015VQA: Visual Question Answeringcited by 2,506
  11. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,141
  12. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,080
38 more
  1. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,919
  2. 2019Towards VQA Models That Can Readcited by 1,718
  3. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,479
  4. 2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,457
  5. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,402
  6. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,317
  7. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,239
  8. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,216
  9. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,121
  10. 2024MiniCPM-V: A GPT-4V Level MLLM on Your Phonecited by 1,079
  11. 2016Modeling Context in Referring Expressionscited by 1,063
  12. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,062
  13. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captionscited by 1,005
  14. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 858
  15. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMscited by 838
  16. 2016A Diagram Is Worth A Dozen Imagescited by 804
  17. 2023Multimodal Chain-of-Thought Reasoning in Language Modelscited by 752
  18. 2024DeepSeek-VL: Towards Real-World Vision-Language Understandingcited by 733
  19. 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 710
  20. 2019MathQA: Towards Interpretable Math Word Problem Solving with Operation-Based Formalismscited by 662
  21. 2023HallusionBench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Modelscited by 583
  22. 2024MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmarkcited by 512
  23. 2020The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memescited by 490
  24. 2024Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Researchcited by 483
  25. 2023OCRBench: On the Hidden Mystery of OCR in Large Multimodal Modelscited by 476
  26. 2024VLMEvalKit: An Open-Source Toolkit for Evaluating Large Multi-Modality Modelscited by 455
  27. 2021InfographicVQAcited by 437
  28. 2021OCR-free Document Understanding Transformercited by 407
  29. 2024Hallucination of Multimodal Large Language Models: A Surveycited by 388
  30. 2021Inter-GPS: Interpretable Geometry Problem Solving with Formal Language and Symbolic Reasoningcited by 383
  31. 2016Visual Dialogcited by 377
  32. 2019KonIQ-10k: An ecologically valid database for deep learning of blind image quality assessmentcited by 355
  33. 2022Dynamic Prompt Learning via Policy Gradient for Semi-structured Mathematical Reasoningcited by 350
  34. 2019FUNSD: A Dataset for Form Understanding in Noisy Scanned Documentscited by 349
  35. 2024Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothingcited by 346
  36. 2018DVQA: Understanding Data Visualizations via Question Answeringcited by 340
  37. 2023PMC-VQA: Visual Instruction Tuning for Medical Visual Question Answeringcited by 338
  38. 2019Scene Text Visual Question Answeringcited by 331

50 most cited of 124 All 124 (JSON)

Built on by

Cited by year

Citing papers per year: 2025: 1.

1 paper.

  1. 2025Modality Curation: Building Universal Embeddings for Advanced Multimodal Information Retrievalcited by 26

Similar, not cited

Close in wording, no citation either way

  1. 202620/20 Vision Language Models: A Prescription for Better VLMs through Data Curation Alone
  2. 2026DataComp-VLM: Improved Open Datasets for Vision-Language Models
  3. 2025Scalable Vision Language Model Training via High Quality Data Curation
  4. 2025Trust the Model: Compact VLMs as In-Context Judges for Image-Text Data Quality
  5. 2024VILA2: VILA Augmented VILA
  6. 2023CiT: Curation in Training for Effective Vision-Language Data
  7. 2024Prismatic VLMs: Investigating the Design Space of Visually-Conditioned Language Models
  8. 2025A Survey of State of the Art Large Vision Language Models: Alignment, Benchmark, Evaluations and Challenges