mlsanityData to 7 Oct 2026

MM-GEN: Enhancing Task Performance Through Targeted Multimodal Data Curation

arXiv 2501.04155 ↗First posted 7 Jan 2025cs.CV cs.CL cs.LGcited by 4

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Vision-language models (VLMs) are highly effective but often underperform on specialized tasks; for example, Llava-1.5 struggles with chart and diagram understanding due to scarce task-specific training data. Existing training data, sourced from general-purpose datasets, fails to capture the nuanced details needed for these tasks. We introduce MM-Gen, a scalable method that generates task-specific, high-quality synthetic text for candidate images by leveraging stronger models. MM-Gen employs a three-stage targeted process: partitioning data into subgroups, generating targeted text based on task descriptions, and filtering out redundant and outlier data. Fine-tuning VLMs with data generated by MM-Gen leads to significant performance gains, including 29% on spatial reasoning and 15% on diagram understanding for Llava-1.5 (7B). Compared to human-curated caption data, MM-Gen achieves up to 1.6x better improvements for the original models, proving its effectiveness in enhancing task-specific VLM performance and bridging the gap between general-purpose datasets and specialized requirements. Code available at https://github.com/sjoshi804/MM-Gen.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 38 of its 57 references that are in the map. First posted on arXiv 7 Jan 2025.

Builds on

38 of 57 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024BLINK: Multimodal Large Language Models Can See but Not Perceive18 Apr
  2. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captions21 Nov

Other references · 36

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2024The Llama 3 Herd of Modelscited by 14,192
  4. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  5. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  6. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  7. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,449
  8. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
  9. 2016A Diagram Is Worth A Dozen Imagescited by 793
  10. 2018An Empirical Study of Example Forgetting during Deep Neural Network Learningcited by 669
  11. 2023DataComp: In search of the next generation of multimodal datasetscited by 635
  12. 2021Deep Learning on a Data Diet: Finding Important Examples Early in Trainingcited by 529
24 more
  1. 2023TinyStories: How Small Can Language Models Be and Still Speak Coherent English?cited by 425
  2. 2020Dataset Cartography: Mapping and Diagnosing Datasets with Training Dynamicscited by 423
  3. 2019Selection via Proxy: Efficient Data Selection for Deep Learningcited by 337
  4. 2018Not All Samples Are Created Equal: Deep Learning with Importance Samplingcited by 323
  5. 2023SemDeDup: Data-efficient learning at web-scale through semantic deduplicationcited by 295
  6. 2021GRAD-MATCH: Gradient Matching based Data Subset Selection for Efficient Deep Model Trainingcited by 269
  7. 2023What's "up" with vision-language models? Investigating their struggle with spatial reasoningcited by 245
  8. 2023Improving CLIP Training with Language Rewritescited by 216
  9. 2023D4: Improving LLM Pretraining via Document De-Duplication and Diversificationcited by 146
  10. 2023M3IT: A Large-Scale Dataset towards Multi-Modal Multilingual Instruction Tuningcited by 126
  11. 2016Show and Tell: Lessons learned from the 2015 MSCOCO Image Captioning Challengecited by 116
  12. 2024Is A Picture Worth A Thousand Words? Delving Into Spatial Reasoning for Vision Language Modelscited by 111
  13. 2023Improving Multimodal Datasets with Image Captioningcited by 93
  14. 2023VeCLIP: Improving CLIP Training via Visual-enriched Captionscited by 68
  15. 2022Adaptive Second Order Coresets for Data-efficient Machine Learningcited by 60
  16. 2023T-MARS: Improving Visual Representations by Circumventing Text Feature Learningcited by 43
  17. 2023FuseCap: Leveraging Large Language Models for Enriched Fused Image Captionscited by 33
  18. 2024Data curation via joint example selection further accelerates multimodal learningcited by 32
  19. 2023Data-Efficient Contrastive Self-supervised Learning: Most Beneficial Examples for Supervised Learning Contribute the Leastcited by 22
  20. 2024Multimodal Self-Instruct: Synthetic Abstract Image and Visual Reasoning Instruction Using Language Modelcited by 21
  21. 2024Data-Efficient Contrastive Language-Image Pretraining: Prioritizing Data Quality over Quantitycited by 19
  22. 2024Eureka: Evaluating and Understanding Large Foundation Modelscited by 18
  23. 2024Revisit Large-Scale Image-Caption Data in Pre-training Multimodal Foundation Modelscited by 14
  24. 2023Decoding Data Quality via Synthetic Corruptions: Embedding-guided Pruning of Code Datacited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 3.

4 papers.

  1. 2026DatBench: Discriminative, Faithful, and Efficient VLM Evaluationscited by 7
  2. 202620/20 Vision Language Models: A Prescription for Better VLMs through Data Curation Alonecited by 2
  3. 2025Test-Time Warmup for Multimodal Large Language Modelscited by 0
  4. 2026Unlocking UML Class Diagram Understanding in Vision Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024MM1.5: Methods, Analysis & Insights from Multimodal LLM Fine-tuning
  2. 2025Synthetic Stimuli, Real Gains: Rethinking VLM Fine-Tuning Through Fully Controlled Data Generation
  3. 2024MMGenBench: Fully Automatically Evaluating LMMs from the Text-to-Image Generation Perspective
  4. 2024MMT-Bench: A Comprehensive Multimodal Benchmark for Evaluating Large Vision-Language Models Towards Multitask AGI
  5. 2024ProVision: Programmatically Scaling Vision-centric Instruction Data for Multimodal Language Models
  6. 2024Task Me Anything
  7. 2024MM-Vet v2: A Challenging Benchmark to Evaluate Large Multimodal Models for Integrated Capabilities
  8. 2025MM-R1: Unleashing the Power of Unified Multimodal Large Language Models for Personalized Image Generation