mlsanityData to 7 Oct 2026

A Survey on Multimodal Large Language Models

arXiv 2306.13549 ↗First posted 23 Jun 2023cs.CV cs.AI cs.CLcited by 898

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Recently, Multimodal Large Language Model (MLLM) represented by GPT-4V has been a new rising research hotspot, which uses powerful Large Language Models (LLMs) as a brain to perform multimodal tasks. The surprising emergent capabilities of MLLM, such as writing stories based on images and OCR-free math reasoning, are rare in traditional multimodal methods, suggesting a potential path to artificial general intelligence. To this end, both academia and industry have endeavored to develop MLLMs that can compete with or even better than GPT-4V, pushing the limit of research at a surprising speed. In this paper, we aim to trace and summarize the recent progress of MLLMs. First of all, we present the basic formulation of MLLM and delineate its related concepts, including architecture, training strategy and data, as well as evaluation. Then, we introduce research topics about how MLLMs can be extended to support more granularity, modalities, languages, and scenarios. We continue with multimodal hallucination and extended techniques, including Multimodal ICL (M-ICL), Multimodal CoT (M-CoT), and LLM-Aided Visual Reasoning (LAVR). To conclude the paper, we discuss existing challenges and point out promising research directions. In light of the fact that the era of MLLM has only just begun, we will keep updating this survey and hope it can inspire more research. An associated GitHub link collecting the latest papers is available at https://github.com/BradyFU/Awesome-Multimodal-Large-Language-Models.

Key facts. Cited by 898 papers in the map (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 127 of its 251 references that are in the map. First posted on arXiv 23 Jun 2023.

Builds on

127 of 251 references.

Closest ancestors · 5

Same direction, newest first

  1. 2023LVLM-eHub: A Comprehensive Evaluation Benchmark for Large Vision-Language Models15 Jun
  2. 2023Macaw-LLM: Multi-Modal Language Modeling with Image, Audio, Video, and Text Integration15 Jun
  3. 2023SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilities18 May
  4. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning11 May
  5. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuning5 May

Other references · 122

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  5. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  6. 2023GPT-4 Technical Reportcited by 18,132
  7. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  8. 2022Training language models to follow instructions with human feedbackcited by 15,712
  9. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  10. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  11. 2023Segment Anythingcited by 8,158
  12. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
38 more
  1. 2023Visual Instruction Tuningcited by 7,792
  2. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  3. 2020End-to-End Object Detection with Transformerscited by 7,384
  4. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  5. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  6. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  7. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  8. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  9. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  10. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  11. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  12. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  13. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  14. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  15. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  16. 2020Learning to summarize from human feedbackcited by 2,562
  17. 2023A Survey of Large Language Modelscited by 2,537
  18. 2015VQA: Visual Question Answeringcited by 2,489
  19. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  20. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  21. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  22. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  23. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,722
  24. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
  25. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
  26. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
  27. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  28. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,461
  29. 2022Least-to-Most Prompting Enables Complex Reasoning in Large Language Modelscited by 1,397
  30. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,352
  31. 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,310
  32. 2022DINO: DETR with Improved DeNoising Anchor Boxes for End-to-End Object Detectioncited by 1,306
  33. 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Daycited by 1,287
  34. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
  35. 2023HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Facecited by 1,173
  36. 2022Reproducible scaling laws for contrastive language-image learningcited by 1,163
  37. 2023ImageBind: One Embedding Space To Bind Them Allcited by 1,162
  38. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,122

50 most cited of 122 All 122 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 76; 2024: 273; 2025: 315; 2026: 234.

898 papers.

Sort:

By area:

  1. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,669
  2. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,422
  3. 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysiscited by 1,375
  4. 2024Show-o: One Single Transformer to Unify Multimodal Understanding and Generationcited by 734
  5. 2023Time-LLM: Time Series Forecasting by Reprogramming Large Language Modelscited by 680
  6. 2023A Comprehensive Overview of Large Language Modelscited by 639
  7. 2024A Survey on the Memory Mechanism of Large Language Model based Agentscited by 622
  8. 2023HallusionBench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Modelscited by 572
  9. 2023MovieChat: From Dense Token to Sparse Memory for Long Video Understandingcited by 536
  10. 2024MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmarkcited by 494
  11. 2024LLaVA-PruMerge: Adaptive Token Reduction for Efficient Large Multimodal Modelscited by 394
  12. 2024A Survey on Hallucination in Large Vision-Language Modelscited by 352
38 more
  1. 2023PointLLM: Empowering Large Language Models to Understand Point Cloudscited by 326
  2. 2024MoE-LLaVA: Mixture of Experts for Large Vision-Language Modelscited by 298
  3. 2023InternLM-XComposer: A Vision-Language Large Model for Advanced Text-image Comprehension and Compositioncited by 297
  4. 2023A Survey on Multimodal Large Language Models for Autonomous Drivingcited by 288
  5. 2024MME-RealWorld: Could Your Multimodal LLM Challenge High-Resolution Real-World Scenarios that are Difficult for Humans?cited by 284
  6. 2025MMaDA: Multimodal Large Diffusion Language Modelscited by 241
  7. 2023Woodpecker: Hallucination Correction for Multimodal Large Language Modelscited by 223
  8. 2023Exploring Collaboration Mechanisms for LLM Agents: A Social Psychology Viewcited by 221
  9. 2024CogVLM2: Visual Language Models for Image and Video Understandingcited by 214
  10. 2023LL3DA: Visual Interactive Instruction Tuning for Omni-3D Understanding, Reasoning, and Planningcited by 206
  11. 2023Video Understanding with Large Language Models: A Surveycited by 203
  12. 2023Multimodal Large Language Models: A Surveycited by 193
  13. 2023Large Language Models for Generative Information Extraction: A Surveycited by 183
  14. 2023How Robust is Google's Bard to Adversarial Image Attacks?cited by 177
  15. 2024VITA: Towards Open-Source Interactive Omni Multimodal LLMcited by 172
  16. 2024Ovis: Structural Embedding Alignment for Multimodal Large Language Modelcited by 167
  17. 2024GME: Improving Universal Multimodal Retrieval by Multimodal LLMscited by 166
  18. 2024Safety Fine-Tuning at (Almost) No Cost: A Baseline for Vision Large Language Modelscited by 160
  19. 2024Large Language Model-Brained GUI Agents: A Surveycited by 160
  20. 2024A Comprehensive Study of Knowledge Editing for Large Language Modelscited by 157
  21. 2024OmniACT: A Dataset and Benchmark for Enabling Multimodal Generalist Autonomous Agents for Desktop and Webcited by 156
  22. 2025UniTok: A Unified Tokenizer for Visual Generation and Understandingcited by 148
  23. 2024LLM Inference Unveiled: Survey and Roofline Model Insightscited by 147
  24. 2024Devils in Middle Layers of Large Vision-Language Models: Interpreting, Detecting and Mitigating Object Hallucinations via Attention Lenscited by 139
  25. 2024Eagle: Exploring The Design Space for Multimodal LLMs with Mixture of Encoderscited by 137
  26. 2024The Revolution of Multimodal Large Language Models: A Surveycited by 134
  27. 2023Large Language Models for Robotics: A Surveycited by 132
  28. 2023Silkie: Preference Distillation for Large Visual Language Modelscited by 128
  29. 2024The Emerged Security and Privacy of LLM Agent: A Survey with Case Studiescited by 127
  30. 2025See What You Are Told: Visual Attention Sink in Large Multimodal Modelscited by 127
  31. 2023Large language models in healthcare and medical domain: A reviewcited by 126
  32. 2023HalluciDoctor: Mitigating Hallucinatory Toxicity in Visual Instruction Datacited by 124
  33. 2024Visual In-Context Learning for Large Vision-Language Modelscited by 124
  34. 2023A Survey of Large Language Models in Medicine: Progress, Application, and Challengecited by 123
  35. 2024E5-V: Universal Embeddings with Multimodal Large Language Modelscited by 123
  36. 2024Images are Achilles' Heel of Alignment: Exploiting Visual Vulnerabilities for Jailbreaking Multimodal Large Language Modelscited by 121
  37. 2023MobileVLM : A Fast, Strong and Open Vision Language Assistant for Mobile Devicescited by 120
  38. 2023Investigating the Catastrophic Forgetting in Multimodal Large Language Modelscited by 119

50 most cited of 898 All 898 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024A Survey on Evaluation of Multimodal Large Language Models
  2. 2024Exploring the Reasoning Abilities of Multimodal Large Language Models (MLLMs): A Comprehensive Survey on Emerging Trends in Multimodal Reasoning
  3. 2024A Survey of Multimodal Large Language Model from A Data-centric Perspective
  4. 2024A Survey on Benchmarks of Multimodal Large Language Models
  5. 2025Benchmarking Large and Small MLLMs
  6. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models
  7. 2025Large language models for artificial general intelligence (AGI): A survey of foundational principles and approaches
  8. 2025On Path to Multimodal Generalist: General-Level and General-Bench