mlsanityData to 7 Oct 2026

The Synergy between Data and Multi-Modal Large Language Models: A Survey from Co-Development Perspective

arXiv 2407.08583 ↗First posted 11 Jul 2024cs.AI cs.CV cs.LGcited by 14

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

The rapid development of large language models (LLMs) has been witnessed in recent years. Based on the powerful LLMs, multi-modal LLMs (MLLMs) extend the modality from text to a broader spectrum of domains, attracting widespread attention due to the broader range of application scenarios. As LLMs and MLLMs rely on vast amounts of model parameters and data to achieve emergent capabilities, the importance of data is receiving increasingly widespread attention and recognition. Tracing and analyzing recent data-oriented works for MLLMs, we find that the development of models and data is not two separate paths but rather interconnected. On the one hand, vaster and higher-quality data contribute to better performance of MLLMs; on the other hand, MLLMs can facilitate the development of data. The co-development of multi-modal data and MLLMs requires a clear view of 1) at which development stages of MLLMs specific data-centric approaches can be employed to enhance certain MLLM capabilities, and 2) how MLLMs, utilizing those capabilities, can contribute to multi-modal data in specific roles. To promote the data-model co-development for MLLM community, we systematically review existing works related to MLLMs from the data-model co-development perspective. A regularly maintained project associated with this survey is accessible at https://github.com/modelscope/data-juicer/blob/main/docs/awesome_llm_data.md.

Key facts. Cited by 14 papers in the map (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 258 of its 296 references that are in the map. First posted on arXiv 11 Jul 2024.

Builds on

258 of 296 references.

Closest ancestors · 39

Same direction, newest first

  1. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs24 Jun
  2. 2024AIM: Let Any Multi-modal Large Language Models Embrace Efficient In-Context Learning11 Jun
  3. 2024A Survey of Multimodal Large Language Model from A Data-centric Perspective26 May
  4. 2024AlignGPT: Multi-modal Large Language Models with Adaptive Alignment Capability23 May
  5. 2024Efficient Multimodal Large Language Models: A Survey17 May
  6. 2024List Items One by One: A New Data Source and Learning Paradigm for Multimodal LLMs25 Apr
  7. 2024MMT-Bench: A Comprehensive Multimodal Benchmark for Evaluating Large Vision-Language Models Towards Multitask AGI24 Apr
  8. 2024ImplicitAVE: An Open-Source Dataset and Multimodal LLMs Benchmark for Implicit Attribute Value Extraction24 Apr
  9. 2024Multimodal Large Language Model is a Human-Aligned Annotator for Text-to-Image Generation23 Apr
  10. 2024BLINK: Multimodal Large Language Models Can See but Not Perceive18 Apr
  11. 2024Draw-and-Understand: Leveraging Visual Prompts to Enable MLLMs to Comprehend What You Want29 Mar
  12. 2024Towards Multimodal In-Context Learning for Vision & Language Models19 Mar
27 more
  1. 2024Finetuned Multimodal Language Models Are High-Quality Image-Text Data Filters5 Mar
  2. 2024The Revolution of Multimodal Large Language Models: A Survey19 Feb
  3. 2024AnyGPT: Unified Multimodal LLM with Discrete Sequence Modeling19 Feb
  4. 2024Vision-Flan: Scaling Human-Labeled Tasks in Visual Instruction Tuning18 Feb
  5. 2024Efficient Multimodal Learning from Data-centric Perspective18 Feb
  6. 2024SPHINX-X: Scaling Data and Parameters for a Family of Multi-modal Large Language Models8 Feb
  7. 2024MLLM-as-a-Judge: Assessing Multimodal LLM-as-a-Judge with Vision-Language Benchmark7 Feb
  8. 2024MM-LLMs: Recent Advances in MultiModal Large Language Models24 Jan
  9. 2024Exploring the Reasoning Abilities of Multimodal Large Language Models (MLLMs): A Comprehensive Survey on Emerging Trends in Multimodal Reasoning10 Jan
  10. 2023Generative Multimodal Models are In-Context Learners20 Dec
  11. 2023Silkie: Preference Distillation for Large Visual Language Models17 Dec
  12. 2023MLLM-Bench: Evaluating Multimodal LLMs with Per-sample Criteria23 Nov
  13. 2023Multimodal Large Language Models: A Survey22 Nov
  14. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captions21 Nov
  15. 2023mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaboration7 Nov
  16. 2023What Makes for Good Visual Instructions? Synthesizing Complex Visual Reasoning Instructions for Visual Instruction Tuning2 Nov
  17. 2023Set-of-Mark Prompting Unleashes Extraordinary Visual Grounding in GPT-4V17 Oct
  18. 2023Aligning Large Multimodal Models with Factually Augmented RLHF25 Sep
  19. 2023MMICL: Empowering Vision-language Model with Multi-Modal In-Context Learning14 Sep
  20. 2023Sight Beyond Text: Multi-Modal Training Enhances LLMs in Truthfulness and Ethics13 Sep
  21. 2023NExT-GPT: Any-to-Any Multimodal LLM11 Sep
  22. 2023MM-LIMA: Less Is More for Alignment in Multi-Modal Datasets23 Aug
  23. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehension30 Jul
  24. 2023BuboGPT: Enabling Visual Grounding in Multi-Modal LLMs17 Jul
  25. 2023Visual Instruction Tuning with Polite Flamingo3 Jul
  26. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magic27 Jun
  27. 2023A Survey on Multimodal Large Language Models23 Jun

Other references · 219

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  4. 2023GPT-4 Technical Reportcited by 18,132
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  6. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  7. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  8. 2023Segment Anythingcited by 8,158
  9. 2023Visual Instruction Tuningcited by 7,792
  10. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  11. 2020Scaling Laws for Neural Language Modelscited by 6,411
  12. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
38 more
  1. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  2. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  3. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  4. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  5. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  6. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  7. 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,591
  8. 2023A Survey of Large Language Modelscited by 2,537
  9. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasetscited by 2,526
  10. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  11. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  12. 2023VBench: Comprehensive Benchmark Suite for Video Generative Modelscited by 1,755
  13. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
  14. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,449
  15. 2023ImageBind: One Embedding Space To Bind Them Allcited by 1,162
  16. 2023MVBench: A Comprehensive Multi-modal Video Understanding Benchmarkcited by 1,086
  17. 2023LIMA: Less Is More for Alignmentcited by 1,080
  18. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,075
  19. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,047
  20. 2023VideoChat: Chat-Centric Video Understandingcited by 851
  21. 2023Vision-Language Models for Vision Tasks: A Surveycited by 834
  22. 2022Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentationcited by 707
  23. 2023Time-LLM: Time Series Forecasting by Reprogramming Large Language Modelscited by 680
  24. 2023DataComp: In search of the next generation of multimodal datasetscited by 635
  25. 2017Deep Mutual Learningcited by 621
  26. 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedbackcited by 615
  27. 2023LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Modelcited by 602
  28. 2024MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems?cited by 573
  29. 2022Beyond neural scaling laws: beating power law scaling via data pruningcited by 519
  30. 2023MM-REACT: Prompting ChatGPT for Multimodal Reasoning and Actioncited by 511
  31. 2022Visual Programming: Compositional visual reasoning without trainingcited by 507
  32. 2023InternVid: A Large-scale Video-Text Dataset for Multimodal Understanding and Generationcited by 488
  33. 2023OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocationcited by 468
  34. 2024Sora: A Review on Background, Technology, Limitations, and Opportunities of Large Vision Modelscited by 467
  35. 2023Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuningcited by 439
  36. 2023RLHF-V: Towards Trustworthy MLLMs via Behavior Alignment from Fine-grained Correctional Human Feedbackcited by 404
  37. 2022Multimodal Learning with Transformers: A Surveycited by 399
  38. 2024Panda-70M: Captioning 70M Videos with Multiple Cross-Modality Teacherscited by 389

50 most cited of 219 All 219 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 5; 2025: 7; 2026: 2.

14 papers.

Sort:

By area:

  1. 2024Img-Diff: Contrastive Data Synthesis for Multimodal Large Language Modelscited by 17
  2. 2025Diversity as a Reward: Fine-Tuning LLMs on a Mixture of Domain-Undetermined Datacited by 13
  3. 2025Explainable Action Form Assessment by Exploiting Multimodal Chain-of-Thoughts Reasoningcited by 13
  4. 2024Data-Juicer 2.0: Cloud-Scale Adaptive Data Processing for and with Foundation Modelscited by 11
  5. 2024Data-Juicer Sandbox: A Feedback-Driven Suite for Multimodal Data-Model Co-developmentcited by 10
  6. 2024HumanVBench: Probing Human-Centric Video Understanding in MLLMs with Automatically Synthesized Benchmarkscited by 10
  7. 2024Federated Data-Efficient Instruction Tuning for Large Language Modelscited by 9
  8. 2026Can LLMs Clean Up Your Mess? A Survey of Application-Ready Data Preparation with LLMscited by 9
  9. 2025MindGYM: What Matters in Question Synthesis for Thinking-Centric Fine-Tuning?cited by 6
  10. 2025DetailMaster: Can Your Text-to-Image Model Handle Long Prompts?cited by 5
  11. 2025A Survey of LLM × DATAcited by 4
  12. 2025Learning from Reasoning Failures via Synthetic Data Generationcited by 2
2 more
  1. 2025MAJORScore: A Novel Metric for Evaluating Multimodal Relevance via Joint Representationcited by 0
  2. 2026Near-Oracle KV Selection via Pre-hoc Sparsity for Long-Context Inferencecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Multilingual Large Language Models: A Systematic Survey
  2. 2024Multilingual Large Language Model: A Survey of Resources, Taxonomy and Frontiers
  3. 2024A Survey on Multilingual Large Language Models: Corpora, Alignment, and Bias
  4. 2024Responsible Multilingual Large Language Models: A Survey of Development, Applications, and Societal Impact
  5. 2025LLLMs: A Data-Driven Survey of Evolving Research on Limitations of Large Language Models
  6. 2023Macaw-LLM: Multi-Modal Language Modeling with Image, Audio, Video, and Text Integration
  7. 2024Survey of different Large Language Model Architectures: Trends, Benchmarks, and Challenges
  8. 2023A Comprehensive Overview of Large Language Models