mlsanityData to 14 Oct 2026

Survey of Large Multimodal Model Datasets, Application Categories and Taxonomy

arXiv 2412.17759 ↗First posted 23 Dec 2024cs.AI cs.CV cs.LGcited by 17

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Multimodal learning, a rapidly evolving field in artificial intelligence, seeks to construct more versatile and robust systems by integrating and analyzing diverse types of data, including text, images, audio, and video. Inspired by the human ability to assimilate information through many senses, this method enables applications such as text-to-video conversion, visual question answering, and image captioning. Recent developments in datasets that support multimodal language models (MLLMs) are highlighted in this overview. Large-scale multimodal datasets are essential because they allow for thorough testing and training of these models. With an emphasis on their contributions to the discipline, the study examines a variety of datasets, including those for training, domain-specific tasks, and real-world applications. It also emphasizes how crucial benchmark datasets are for assessing models' performance in a range of scenarios, scalability, and applicability. Since multimodal learning is always changing, overcoming these obstacles will help AI research and applications reach new heights.

Key facts. Cited by 17 papers in the map (data through 14 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 42 of its 63 references that are in the map. First posted on arXiv 23 Dec 2024.

Builds on

42 of 63 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024A Survey of Multimodal Large Language Model from A Data-centric Perspective26 May
  2. 2024List Items One by One: A New Data Source and Learning Paradigm for Multimodal LLMs25 Apr
  3. 2024Peacock: A Family of Arabic Multimodal Large Language Models and Benchmarks1 Mar
  4. 2024MM-LLMs: Recent Advances in MultiModal Large Language Models24 Jan

Other references · 38

  1. 2023Visual Instruction Tuningcited by 7,869
  2. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,523
  3. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
  4. 2015ShapeNet: An Information-Rich 3D Model Repositorycited by 2,897
  5. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,462
  6. 2019Common Voice: A Massively-Multilingual Speech Corpuscited by 1,384
  7. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,216
  8. 2019HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clipscited by 950
  9. 2023Vision-Language Models for Vision Tasks: A Surveycited by 839
  10. 2019ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Taskscited by 783
  11. 2017VoxCeleb: a large-scale speaker identification datasetcited by 732
  12. 2023A Comprehensive Overview of Large Language Modelscited by 648
26 more
  1. 2018MELD: A Multimodal Multi-Party Dataset for Emotion Recognition in Conversationscited by 643
  2. 2023InternVid: A Large-scale Video-Text Dataset for Multimodal Understanding and Generationcited by 497
  3. 2018TVQA: Localized, Compositional Video Question Answeringcited by 467
  4. 2024OpenVid-1M: A Large-Scale High-Quality Dataset for Text-to-video Generationcited by 341
  5. 2022SQA3D: Situated Question Answering in 3D Scenescited by 314
  6. 2023MIMIC-IT: Multi-Modal In-Context Instruction Tuningcited by 267
  7. 2018SpaceNet: A Remote Sensing Dataset and Challenge Seriescited by 179
  8. 2023LAMM: Language-Assisted Multi-Modal Instruction-Tuning Dataset, Framework, and Benchmarkcited by 169
  9. 2023SVIT: Scaling up Visual Instruction Tuningcited by 162
  10. 2024OmniACT: A Dataset and Benchmark for Enabling Multimodal Generalist Autonomous Agents for Desktop and Webcited by 157
  11. 2019CATER: A diagnostic dataset for Compositional Actions and TEmporal Reasoningcited by 137
  12. 2018AISHELL-2: Transforming Mandarin ASR Research Into Industrial Scalecited by 128
  13. 2022Foundations and Trends in Multimodal Machine Learning: Principles, Challenges, and Open Questionscited by 116
  14. 2023Retrieving Multimodal Information for Augmented Generation: A Surveycited by 85
  15. 2024Vript: A Video Is Worth Thousands of Wordscited by 71
  16. 2022M3ED: Multi-modal Multi-scene Multi-label Emotional Dialogue Databasecited by 50
  17. 2023Charting New Territories: Exploring the Geographic and Geospatial Capabilities of Multimodal LLMscited by 34
  18. 2017See, Hear, and Read: Deep Aligned Representationscited by 32
  19. 2024PathGen-1.6M: 1.6 Million Pathology Image-text Pairs Generation through Multi-agent Collaborationcited by 28
  20. 2023Multimodality Representation Learning: A Survey on Evolution, Pretraining and Its Applicationscited by 27
  21. 2024LVLM-Interpret: An Interpretability Tool for Large Vision-Language Modelscited by 24
  22. 2024MMSci: A Dataset for Graduate-Level Multi-Discipline Multimodal Scientific Understandingcited by 24
  23. 2022Does a Technique for Building Multimodal Representation Matter? -- Comparative Analysiscited by 22
  24. 2023MMSum: A Dataset for Multimodal Summarization and Thumbnail Generation of Videoscited by 12
  25. 2023Images in Language Space: Exploring the Suitability of Large Language Models for Vision & Language Taskscited by 5
  26. 2024EmotionCaps: Enhancing Audio Captioning Through Emotion-Augmented Data Generationcited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 10; 2026: 6.

17 papers.

Sort:

By area:

  1. 2025Position: Multimodal Large Language Models Can Significantly Advance Scientific Reasoningcited by 33
  2. 2025Tokenization Matters: Improving Zero-Shot NER for Indic Languagescited by 12
  3. 2024MVTamperBench: Evaluating Robustness of Vision-Language Modelscited by 11
  4. 2025Aligning LLMs for Multilingual Consistency in Enterprise Applicationscited by 11
  5. 2025FS-DAG: Few Shot Domain Adapting Graph Networks for Visually Rich Document Understandingcited by 10
  6. 2025Clinical QA 2.0: Multi-Task Learning for Answer Extraction and Categorizationcited by 9
  7. 2025Can LLMs Narrate Tabular Data? An Evaluation Framework for Natural Language Representations of Text-to-SQL System Outputscited by 7
  8. 2026IndicJR: A Judge-Free Benchmark of Jailbreak Robustness in South Asian Languagescited by 5
  9. 2025Multimodal Representation Learning and Fusioncited by 3
  10. 2025RCI: A Score for Evaluating Global and Local Reasoning in Multimodal Benchmarkscited by 2
  11. 2025FlexDoc: Parameterized Sampling for Diverse Multilingual Synthetic Documents for Training Document Understanding Modelscited by 2
  12. 2026LLM-Guided Lifecycle-Aware Clustering of Multi-Turn Customer Support Conversationscited by 2
5 more
  1. 2026Do Image-Text Metrics Respect Semantic Invariances?cited by 2
  2. 2025PCRI: Measuring Context Robustness in Multimodal Models for Enterprise Applicationscited by 1
  3. 2026Scaling-Aware Data Selection for End-to-End Autonomous Driving Systemscited by 1
  4. 2026Dynamic Adversarial Reinforcement Learning for Robust Multimodal Large Language Modelscited by 0
  5. 2026Test-Time Coverage: Test-Conditioned Data Curation for Deployment-Aware Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Multimodal Large Language Models and Tunings: Vision, Language, Sensors, Audio, and Beyond
  2. 2024A Comprehensive Review of Multimodal Large Language Models: Performance and Challenges Across Different Tasks
  3. 2023Multimodal Large Language Models: A Survey
  4. 2024Explainable and Interpretable Multimodal Large Language Models: A Comprehensive Survey
  5. 2024A Comprehensive Survey and Guide to Multimodal Large Language Models in Vision-Language Tasks
  6. 2025Multimodal Large Language Models for Image, Text, and Speech Data Augmentation: A Survey
  7. 2024The Revolution of Multimodal Large Language Models: A Survey
  8. 2024A Survey on Benchmarks of Multimodal Large Language Models