mlsanityData to 14 Oct 2026

A Survey of Multimodal Large Language Model from A Data-centric Perspective

arXiv 2405.16640 ↗First posted 26 May 2024cs.AI cs.CL cs.CVcited by 75

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Multimodal large language models (MLLMs) enhance the capabilities of standard large language models by integrating and processing data from multiple modalities, including text, vision, audio, video, and 3D environments. Data plays a pivotal role in the development and refinement of these models. In this survey, we comprehensively review the literature on MLLMs from a data-centric perspective. Specifically, we explore methods for preparing multimodal data during the pretraining and adaptation phases of MLLMs. Additionally, we analyze the evaluation methods for the datasets and review the benchmarks for evaluating MLLMs. Our survey also outlines potential future research directions. This work aims to provide researchers with a detailed understanding of the data-driven aspects of MLLMs, fostering further exploration and innovation in this field.

Key facts. Cited by 75 papers in the map (data through 14 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 203 of its 286 references that are in the map. First posted on arXiv 26 May 2024.

Builds on

203 of 286 references.

Closest ancestors · 20

Same direction, newest first

  1. 2024MM1: Methods, Analysis & Insights from Multimodal LLM Pre-training14 Mar
  2. 2024Finetuned Multimodal Language Models Are High-Quality Image-Text Data Filters5 Mar
  3. 2024Efficient Multimodal Learning from Data-centric Perspective18 Feb
  4. 2024MLLM-as-a-Judge: Assessing Multimodal LLM-as-a-Judge with Vision-Language Benchmark7 Feb
  5. 2024Variance Alignment Score: A Simple But Tough-to-Beat Data Selection Method for Multimodal Contrastive Learning3 Feb
  6. 2023Multimodal Large Language Models: A Survey22 Nov
  7. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captions21 Nov
  8. 2023Monkey: Image Resolution and Text Label Are Important Things for Large Multi-modal Models11 Nov
  9. 2023Improved Baselines with Visual Instruction Tuning5 Oct
  10. 2023Aligning Large Multimodal Models with Factually Augmented RLHF25 Sep
  11. 2023NExT-GPT: Any-to-Any Multimodal LLM11 Sep
  12. 2023TouchStone: Evaluating Vision-Language Models by Language Models31 Aug
8 more
  1. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities4 Aug
  2. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehension30 Jul
  3. 2023ChatSpot: Bootstrapping Multimodal LLMs via Precise Referring Instruction Tuning18 Jul
  4. 2023BuboGPT: Enabling Visual Grounding in Multi-Modal LLMs17 Jul
  5. 2023MMBench: Is Your Multi-modal Model an All-around Player?12 Jul
  6. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magic27 Jun
  7. 2023Macaw-LLM: Multi-Modal Language Modeling with Image, Audio, Video, and Text Integration15 Jun
  8. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning11 May

Other references · 183

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  6. 2022Training language models to follow instructions with human feedbackcited by 15,971
  7. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  8. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  9. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  10. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
  11. 2020Scaling Laws for Neural Language Modelscited by 6,586
  12. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
38 more
  1. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  2. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
  3. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  4. 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,117
  5. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,656
  6. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasetscited by 2,581
  7. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
  8. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
  9. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
  10. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,181
  11. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,919
  12. 2019Towards VQA Models That Can Readcited by 1,718
  13. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,614
  14. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,555
  15. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,479
  16. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,471
  17. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,462
  18. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,402
  19. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,317
  20. 2022Text Embeddings by Weakly-Supervised Contrastive Pre-trainingcited by 1,253
  21. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,216
  22. 2016Deep CORAL: Correlation Alignment for Deep Domain Adaptationcited by 1,200
  23. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,177
  24. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,121
  25. 2023MVBench: A Comprehensive Multi-modal Video Understanding Benchmarkcited by 1,109
  26. 2017The "something something" video database for learning and evaluating visual common sensecited by 1,103
  27. 2023LIMA: Less Is More for Alignmentcited by 1,100
  28. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,040
  29. 2020MPNet: Masked and Permuted Pre-training for Language Understandingcited by 981
  30. 2014Deep Visual-Semantic Alignments for Generating Image Descriptionscited by 968
  31. 2019HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clipscited by 950
  32. 2015Generation and Comprehension of Unambiguous Object Descriptionscited by 904
  33. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 897
  34. 2022CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesiscited by 885
  35. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 858
  36. 2023Vision-Language Models for Vision Tasks: A Surveycited by 839
  37. 2016A Diagram Is Worth A Dozen Imagescited by 804
  38. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 766

50 most cited of 183 All 183 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 24; 2025: 30; 2026: 21.

75 papers.

Sort:

By area:

  1. 2024MAmmoTH-VL: Eliciting Multimodal Reasoning with Instruction Tuning at Scalecited by 127
  2. 2025A Survey of Context Engineering for Large Language Modelscited by 126
  3. 2024Recent Advances in Speech Language Models: A Surveycited by 87
  4. 2025Eagle 2: Building Post-Training Data Strategies from Scratch for Frontier Vision-Language Modelscited by 70
  5. 2024MJ-Bench: Is Your Multimodal Reward Model Really a Good Judge for Text-to-Image Generation?cited by 66
  6. 2024A Survey of Mathematical Reasoning in the Era of Multimodal Large Language Model: Benchmark, Method & Challengescited by 55
  7. 2024KeyVideoLLM: Towards Large-scale Video Keyframe Selectioncited by 54
  8. 2024MC-LLaVA: Multi-Concept Personalized Vision-Language Modelcited by 54
  9. 2024LOKI: A Comprehensive Synthetic Data Detection Benchmark using Large Multimodal Modelscited by 51
  10. 2025MM-Verify: Enhancing Multimodal Reasoning with Chain-of-Thought Verificationcited by 39
  11. 2025VisualWebInstruct: Scaling up Multimodal Instruction Data through Web Searchcited by 35
  12. 2025Position: Multimodal Large Language Models Can Significantly Advance Scientific Reasoningcited by 33
38 more
  1. 2025MV-MATH: Evaluating Multimodal Math Reasoning in Multi-Visual Contextscited by 32
  2. 2024Unveiling the Ignorance of MLLMs: Seeing Clearly, Answering Incorrectlycited by 30
  3. 2025Agentic AI Security: Threats, Defenses, Evaluation, and Open Challengescited by 28
  4. 2025DataFlow: An LLM-Driven Framework for Unified Data Preparation and Workflow Automation in the Era of Data-Centric AIcited by 23
  5. 2024Surveying the MLLM Landscape: A Meta-Review of Current Surveyscited by 20
  6. 2024Sample then Identify: A General Framework for Risk Control and Assessment in Multimodal Large Language Modelscited by 19
  7. 2024Survey of Large Multimodal Model Datasets, Application Categories and Taxonomycited by 17
  8. 2024The Synergy between Data and Multi-Modal Large Language Models: A Survey from Co-Development Perspectivecited by 15
  9. 2024Data-Juicer 2.0: Cloud-Scale Adaptive Data Processing for and with Foundation Modelscited by 12
  10. 2025Patch Matters: Training-free Fine-grained Image Caption Enhancement via Local Perceptioncited by 12
  11. 2025SynthRL: Scaling Visual Reasoning with Verifiable Data Synthesiscited by 12
  12. 2024Data-Juicer Sandbox: A Feedback-Driven Suite for Multimodal Data-Model Co-developmentcited by 10
  13. 2024Synth-Empathy: Towards High-Quality Synthetic Empathy Datacited by 10
  14. 2024Are Bigger Encoders Always Better in Vision Large Models?cited by 9
  15. 2024A Review of Multimodal Explainable Artificial Intelligence: Past, Present and Futurecited by 9
  16. 2025Res-Bench: Benchmarking the Robustness of Multimodal Large Language Models to Dynamic Resolution Inputcited by 9
  17. 2026Efficient Inference for Large Vision-Language Models: Bottlenecks, Techniques, and Prospectscited by 9
  18. 2024PAS: Data-Efficient Plug-and-Play Prompt Augmentation Systemcited by 7
  19. 2024SynthVLM: Towards High-Quality and Efficient Synthesis of Image-Caption Datasets for Vision-Language Modelscited by 6
  20. 2025MathDebugger: Detecting and Diagnosing Errors in Synthetic Mathematical Datacited by 6
  21. 2025Scaling Up Biomedical Vision-Language Models: Fine-Tuning, Instruction Tuning, and Multi-Modal Learningcited by 6
  22. 2025Flex-Judge: Text-Only Reasoning Unleashes Zero-Shot Multimodal Evaluatorscited by 6
  23. 2025TiKMiX: Take Data Influence into Dynamic Mixture for Language Model Pre-trainingcited by 6
  24. 2024MathScape: Benchmarking Multimodal Large Language Models in Real-World Mathematical Contextscited by 5
  25. 2025The Unseen Bias: How Norm Discrepancy in Pre-Norm MLLMs Leads to Visual Information Losscited by 5
  26. 2026AgenticRAGTracer: A Hop-Aware Benchmark for Diagnosing Multi-Step Retrieval Reasoning in Agentic RAGcited by 5
  27. 2026DataFlex: A Unified Framework for Data-Centric Dynamic Training of Large Language Modelscited by 5
  28. 2025MaZO: Masked Zeroth-Order Optimization for Multi-Task Fine-Tuning of Large Language Modelscited by 4
  29. 2025RecTok: Reconstruction Distillation along Rectified Flowcited by 4
  30. 2026LongVidSearch: An Agentic Benchmark for Multi-hop Evidence Retrieval Planning in Long Videoscited by 4
  31. 2025Evaluating Sample Utility for Efficient Data Selection by Mimicking Model Weightscited by 3
  32. 2025Towards Multimodal Active Learning: Efficient Learning with Limited Paired Datacited by 3
  33. 2025VideoCompressa: Data-Efficient Video Understanding via Joint Temporal Compression and Spatial Reconstructioncited by 3
  34. 2026Towards Next-Generation LLM Training: From the Data-Centric Perspectivecited by 3
  35. 2026OpenWorldLib: A Unified Codebase and Definition of Advanced World Modelscited by 3
  36. 2024Gradual Learning: Optimizing Fine-Tuning with Partially Mastered Knowledge in Large Language Modelscited by 2
  37. 2024TSPE: Task-Specific Prompt Ensemble for Improved Zero-Shot Audio Classificationcited by 2
  38. 2025ID-Align: RoPE-Conscious Position Remapping for Dynamic High-Resolution Adaptation in Vision-Language Modelscited by 2

50 most cited of 75 All 75 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024A Comprehensive Review of Multimodal Large Language Models: Performance and Challenges Across Different Tasks
  2. 2024The Revolution of Multimodal Large Language Models: A Survey
  3. 2024Personalized Multimodal Large Language Models: A Survey
  4. 2024A Survey on Benchmarks of Multimodal Large Language Models
  5. 2024Efficient Multimodal Large Language Models: A Survey
  6. 2024A Comprehensive Survey and Guide to Multimodal Large Language Models in Vision-Language Tasks
  7. 2023SEED-Bench-2: Benchmarking Multimodal Large Language Models
  8. 2025Multimodal Large Language Models for Text-rich Image Understanding: A Comprehensive Review