mlsanityData to 14 Oct 2026

Multimodal Large Language Models: A Survey

arXiv 2311.13165 ↗First posted 22 Nov 2023cs.AIcited by 195

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

The exploration of multimodal language models integrates multiple data types, such as images, text, language, audio, and other heterogeneity. While the latest large language models excel in text-based tasks, they often struggle to understand and process other data types. Multimodal models address this limitation by combining various modalities, enabling a more comprehensive understanding of diverse data. This paper begins by defining the concept of multimodal and examining the historical development of multimodal algorithms. Furthermore, we introduce a range of multimodal products, focusing on the efforts of major technology companies. A practical guide is provided, offering insights into the technical aspects of multimodal models. Moreover, we present a compilation of the latest algorithms and commonly used datasets, providing researchers with valuable resources for experimentation and evaluation. Lastly, we explore the applications of multimodal models and discuss the challenges associated with their development. By addressing these aspects, this paper aims to facilitate a deeper understanding of multimodal models and their potential in various domains.

Key facts. Cited by 195 papers in the map (data through 14 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 53 of its 76 references that are in the map. First posted on arXiv 22 Nov 2023.

Builds on

53 of 76 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023A Survey on Multimodal Large Language Models23 Jun

Other references · 52

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  6. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  7. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  8. 2023Visual Instruction Tuningcited by 7,869
  9. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
  10. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,355
  11. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  12. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,375
38 more
  1. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  2. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,656
  3. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,375
  4. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,071
  5. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,619
  6. 2019Common Voice: A Massively-Multilingual Speech Corpuscited by 1,384
  7. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,177
  8. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 781
  9. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 766
  10. 2020A Lip Sync Expert Is All You Need for Speech to Lip Generation In The Wildcited by 672
  11. 2023Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Modelscited by 668
  12. 2021Multimodal Few-Shot Learning with Frozen Language Modelscited by 661
  13. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervisioncited by 647
  14. 2017Towards Automatic Learning of Procedures from Web Instructional Videoscited by 619
  15. 2023A Prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPTcited by 619
  16. 2021VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understandingcited by 602
  17. 2023Language Is Not All You Need: Aligning Perception with Language Modelscited by 543
  18. 2023MM-REACT: Prompting ChatGPT for Multimodal Reasoning and Actioncited by 517
  19. 2023MusicLM: Generating Music From Textcited by 448
  20. 2021VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Expertscited by 425
  21. 2021An Empirical Study of GPT-3 for Few-Shot Knowledge-Based VQAcited by 301
  22. 2021An Empirical Study of Training End-to-End Vision-and-Language Transformerscited by 271
  23. 2016Image Captioning with Semantic Attentioncited by 260
  24. 2023Multimodal Deep Learningcited by 258
  25. 2020Byte Pair Encoding is Suboptimal for Language Model Pretrainingcited by 192
  26. 2021MeshTalk: 3D Face Animation from Speech using Cross-Modality Disentanglementcited by 190
  27. 2023mPLUG-2: A Modularized Multi-modal Foundation Model Across Text, Image and Videocited by 154
  28. 2023Large-scale Multi-Modal Pre-trained Models: A Comprehensive Surveycited by 153
  29. 2023Large Language Models for Robotics: A Surveycited by 133
  30. 2022From Images to Textual Prompts: Zero-shot VQA with Frozen Large Language Modelscited by 115
  31. 2020Captioning Images Taken by People Who Are Blindcited by 110
  32. 2020BSL-1K: Scaling up co-articulated sign language recognition using mouthing cuescited by 106
  33. 2022Plug-and-Play VQA: Zero-shot VQA by Conjoining Large Pretrained Models with Zero Trainingcited by 88
  34. 2023AI-Generated Content (AIGC): A Surveycited by 59
  35. 2022A Review on Methods and Applications in Multimodal Deep Learningcited by 29
  36. 2023Model-as-a-Service (MaaS): A Surveycited by 25
  37. 2021Emotion Recognition from Multiple Modalities: Fundamentals and Methodologiescited by 22
  38. 2021Distilled Dual-Encoder Model for Vision-Language Understandingcited by 22

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 71; 2025: 85; 2026: 38.

195 papers.

Sort:

By area:

  1. 2024A Survey on the Memory Mechanism of Large Language Model based Agentscited by 634
  2. 2024Hallucination of Multimodal Large Language Models: A Surveycited by 388
  3. 2024MM-LLMs: Recent Advances in MultiModal Large Language Modelscited by 278
  4. 2024SafeDecoding: Defending against Jailbreak Attacks via Safety-Aware Decodingcited by 247
  5. 2024MeshAnything: Artist-Created Mesh Generation with Autoregressive Transformerscited by 144
  6. 2024A Survey on Large Language Model Acceleration based on KV Cache Managementcited by 131
  7. 2024Mobile Edge Intelligence for Large Language Models: A Contemporary Surveycited by 128
  8. 2024On-Device Language Models: A Comprehensive Reviewcited by 104
  9. 2024Evolutionary Computation in the Era of Large Language Model: Survey and Roadmapcited by 90
  10. 2024Recent Advances in Speech Language Models: A Surveycited by 87
  11. 2024A Survey of Multimodal Large Language Model from A Data-centric Perspectivecited by 75
  12. 2025Large VLM-based Vision-Language-Action Models for Robotic Manipulation: A Surveycited by 69
38 more
  1. 2024Exploring the Frontier of Vision-Language Models: A Survey of Current Methodologies and Future Directionscited by 68
  2. 2024Benchmarking Multimodal Retrieval Augmented Generation with Dynamic VQA Dataset and Self-adaptive Planning Agentcited by 65
  3. 2024A Comprehensive Review of Multimodal Large Language Models: Performance and Challenges Across Different Taskscited by 57
  4. 2024Determinants of LLM-assisted Decision-Makingcited by 56
  5. 2024A Survey of Mathematical Reasoning in the Era of Multimodal Large Language Model: Benchmark, Method & Challengescited by 55
  6. 2024KeyVideoLLM: Towards Large-scale Video Keyframe Selectioncited by 54
  7. 2024Explainable and Interpretable Multimodal Large Language Models: A Comprehensive Surveycited by 52
  8. 2024TopViewRS: Vision-Language Models as Top-View Spatial Reasonerscited by 49
  9. 2025Don't Just Chase "Highlighted Tokens" in MLLMs: Revisiting Visual Holistic Context Retentioncited by 47
  10. 2024Merge, Ensemble, and Cooperate! A Survey on Collaborative Strategies in the Era of Large Language Modelscited by 44
  11. 2024Recent Advances of Multimodal Continual Learning: A Comprehensive Surveycited by 39
  12. 2025MM-Verify: Enhancing Multimodal Reasoning with Chain-of-Thought Verificationcited by 39
  13. 2024Efficient Generation of Targeted and Transferable Adversarial Examples for Vision-Language Models Via Diffusion Modelscited by 38
  14. 2024Private Attribute Inference from Images with Vision-Language Modelscited by 37
  15. 2025Visual Large Language Models for Generalized and Specialized Applicationscited by 36
  16. 2024SceMQA: A Scientific College Entrance Level Multimodal Question Answering Benchmarkcited by 35
  17. 2024X2-DFD: A framework for eXplainable and eXtendable Deepfake Detectioncited by 35
  18. 2024Large Language Models for Education: A Surveycited by 34
  19. 2025MM-CoT:A Benchmark for Probing Visual Chain-of-Thought Reasoning in Multimodal Modelscited by 34
  20. 2025MV-MATH: Evaluating Multimodal Math Reasoning in Multi-Visual Contextscited by 32
  21. 2024Investigating and Mitigating the Multimodal Hallucination Snowballing in Large Vision-Language Modelscited by 31
  22. 2025Shifting AI Efficiency From Model-Centric to Data-Centric Compressioncited by 29
  23. 2024STICKERCONV: Generating Multimodal Empathetic Responses from Scratchcited by 22
  24. 2024The Evolution of Multimodal Model Architecturescited by 21
  25. 2024Visual Reasoning and Multi-Agent Approach in Multimodal Large Language Models (MLLMs): Solving TSP and mTSP Combinatorial Challengescited by 21
  26. 2024LLMs and the Future of Chip Design: Unveiling Security Risks and Building Trustcited by 20
  27. 2024Surveying the MLLM Landscape: A Meta-Review of Current Surveyscited by 20
  28. 2025Hybrid Token Compression for Vision-Language Modelscited by 20
  29. 2024Sample then Identify: A General Framework for Risk Control and Assessment in Multimodal Large Language Modelscited by 19
  30. 2025HumaniBench: A Human-Centric Framework for Large Multimodal Models Evaluationcited by 19
  31. 2025Mixture of Experts (MoE): A Big Data Perspectivecited by 18
  32. 2024Hallucination Detection in Foundation Models for Decision-Making: A Flexible Definition and Review of the State of the Artcited by 17
  33. 2025MME-Emotion: A Holistic Evaluation Benchmark for Emotional Intelligence in Multimodal Large Language Modelscited by 17
  34. 2023Gemini in Reasoning: Unveiling Commonsense in Multimodal Large Language Modelscited by 16
  35. 2024Efficiently Serving Large Multimodal Models Using EPD Disaggregationcited by 16
  36. 2024The Synergy between Data and Multi-Modal Large Language Models: A Survey from Co-Development Perspectivecited by 15
  37. 2025Unveiling the Potential of Diffusion Large Language Model in Controllable Generationcited by 15
  38. 2024Lifelong Knowledge Editing for Vision Language Models with Low-Rank Mixture-of-Expertscited by 14

50 most cited of 195 All 195 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Multimodal Large Language Models for Text-rich Image Understanding: A Comprehensive Review
  2. 2024The Revolution of Multimodal Large Language Models: A Survey
  3. 2024Personalized Multimodal Large Language Models: A Survey
  4. 2023How to Bridge the Gap between Modalities: Survey on Multimodal Large Language Model
  5. 2024A Comprehensive Survey and Guide to Multimodal Large Language Models in Vision-Language Tasks
  6. 2024Multimodal Large Language Models and Tunings: Vision, Language, Sensors, Audio, and Beyond
  7. 2025Multimodal Large Language Models for Image, Text, and Speech Data Augmentation: A Survey
  8. 2024Efficient Multimodal Large Language Models: A Survey