mlsanityData to 7 Oct 2026

Data Processing Techniques for Modern Multimodal Models

arXiv 2407.19180 ↗First posted 27 Jul 2024cs.CVcited by 3

Not on the map. Why?

Abstract (arXiv)

Data processing plays an significant role in current multimodal model training. In this paper. we provide an comprehensive review of common data processing techniques used in modern multimodal model training with a focus on diffusion models and multimodal large language models (MLLMs). We summarized all techniques into four categories: data quality, data quantity, data distribution and data safety. We further present our findings in the choice of data process methods in different type of models. This study aims to provide guidance to multimodal models developers with effective data processing techniques.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 42 of its 55 references that are in the map. First posted on arXiv 27 Jul 2024.

Builds on

42 of 55 references.

References · 42

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  5. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  6. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  7. 2023GPT-4 Technical Reportcited by 18,132
  8. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  9. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  10. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  11. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  12. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,290
30 more
  1. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  2. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  3. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  4. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  5. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  6. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
  7. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
  8. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,114
  9. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captionscited by 1,000
  10. 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 702
  11. 2023Bias and Fairness in Large Language Models: A Surveycited by 626
  12. 2023Textbooks Are All You Needcited by 561
  13. 2023OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Modelscited by 560
  14. 2020TextCaps: a Dataset for Image Captioning with Reading Comprehensioncited by 398
  15. 2021Towards Understanding and Mitigating Social Biases in Language Modelscited by 294
  16. 2021WIT: Wikipedia-based Image Text Dataset for Multimodal Multilingual Machine Learningcited by 285
  17. 2023Emu: Enhancing Image Generation Models Using Photogenic Needles in a Haystackcited by 278
  18. 2021Image Retrieval on Real-life Images with Pre-trained Vision-and-Language Modelscited by 244
  19. 2019Billion-scale semi-supervised learning for image classificationcited by 232
  20. 2023Stable Bias: Analyzing Societal Representations in Diffusion Modelscited by 225
  21. 2023Multimodal C4: An Open, Billion-scale Corpus of Images Interleaved with Textcited by 208
  22. 2023Multimodal Large Language Models: A Surveycited by 193
  23. 2022A Survey of Vision-Language Pre-Trained Modelscited by 177
  24. 2022Can Machines Help Us Answering Question 16 in Datasheets, and In Turn Reflecting on Inappropriate Content?cited by 150
  25. 2020NeuroLogic Decoding: (Un)supervised Neural Text Generation with Predicate Logic Constraintscited by 146
  26. 2023Social Biases through the Text-to-Image Generation Lenscited by 126
  27. 2022A Prompt Array Keeps the Bias Away: Debiasing Vision-Language Models with Adversarial Learningcited by 104
  28. 2022How well can Text-to-Image Generative Models understand Ethical Natural Language Interventions?cited by 92
  29. 2022Multi-modal Alignment using Representation Codebookcited by 63
  30. 2021A survey on VQA_Datasets and Approachescited by 11

Built on by

Cited by year

Citing papers per year: 2024: 1; 2025: 2.

3 papers.

  1. 2024Multimodal Alignment and Fusion: A Surveycited by 73
  2. 2025Multimodal Representation Learning and Fusioncited by 3
  3. 2025OTTER: Open-Tagging via Text-Image Representation for Multi-modal Understandingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024A Survey of Multimodal Large Language Model from A Data-centric Perspective
  2. 2024Survey of Large Multimodal Model Datasets, Application Categories and Taxonomy
  3. 2025Multimodal Large Language Models for Image, Text, and Speech Data Augmentation: A Survey
  4. 2024A Comprehensive Review of Multimodal Large Language Models: Performance and Challenges Across Different Tasks
  5. 2025Empowering Multimodal LLMs with External Tools: A Comprehensive Survey
  6. 2024MME-Survey: A Comprehensive Survey on Evaluation of Multimodal LLMs
  7. 2024Multimodal Large Language Models and Tunings: Vision, Language, Sensors, Audio, and Beyond
  8. 2024MME-RealWorld: Could Your Multimodal LLM Challenge High-Resolution Real-World Scenarios that are Difficult for Humans?