mlsanityData to 7 Oct 2026

The Revolution of Multimodal Large Language Models: A Survey

arXiv 2402.12451 ↗First posted 19 Feb 2024cs.CV cs.AI cs.CLcited by 134

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Connecting text and visual modalities plays an essential role in generative intelligence. For this reason, inspired by the success of large language models, significant research efforts are being devoted to the development of Multimodal Large Language Models (MLLMs). These models can seamlessly integrate visual and textual modalities, while providing a dialogue-based interface and instruction-following capabilities. In this paper, we provide a comprehensive review of recent visual-based MLLMs, analyzing their architectural choices, multimodal alignment strategies, and training techniques. We also conduct a detailed analysis of these models across a wide range of tasks, including visual grounding, image generation and editing, visual understanding, and domain-specific applications. Additionally, we compile and describe training datasets and evaluation benchmarks, conducting comparisons among existing models in terms of performance and computational requirements. Overall, this survey offers a comprehensive overview of the current state of the art, laying the groundwork for future MLLMs.

Key facts. Cited by 134 papers in the map (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 151 of its 253 references that are in the map. First posted on arXiv 19 Feb 2024.

Builds on

151 of 253 references.

Closest ancestors · 35

Same direction, newest first

  1. 2024SPHINX-X: Scaling Data and Parameters for a Family of Multi-modal Large Language Models8 Feb
  2. 2023FoodLMM: A Versatile Food Assistant using Large Multi-modal Model22 Dec
  3. 2023Generative Multimodal Models are In-Context Learners20 Dec
  4. 2023VILA: On Pre-training for Visual Language Models12 Dec
  5. 2023Honeybee: Locality-enhanced Projector for Multimodal LLM11 Dec
  6. 2023CoDi-2: In-Context, Interleaved, and Interactive Any-to-Any Generation30 Nov
  7. 2023LLMGA: Multimodal Large Language Model based Generation Assistant27 Nov
  8. 2023LION : Empowering Multimodal Large Language Model with Dual-Level Visual Knowledge20 Nov
  9. 2023SPHINX: The Joint Mixing of Weights, Tasks, and Visual Embeddings for Multi-modal Large Language Models13 Nov
  10. 2023Monkey: Image Resolution and Text Label Are Important Things for Large Multi-modal Models11 Nov
  11. 2023LLaVA-Plus: Learning to Use Tools for Creating Multimodal Agents9 Nov
  12. 2023mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaboration7 Nov
23 more
  1. 2023Improved Baselines with Visual Instruction Tuning5 Oct
  2. 2023Kosmos-G: Generating Images in Context with Multimodal Large Language Models4 Oct
  3. 2023MiniGPT-5: Interleaved Vision-and-Language Generation via Generative Vokens3 Oct
  4. 2023Pink: Unveiling the Power of Referential Comprehension for Multi-modal LLMs1 Oct
  5. 2023DreamLLM: Synergistic Multimodal Comprehension and Creation20 Sep
  6. 2023NExT-GPT: Any-to-Any Multimodal LLM11 Sep
  7. 2023Scaling Autoregressive Multi-Modal Models: Pretraining and Instruction Tuning5 Sep
  8. 2023TouchStone: Evaluating Vision-Language Models by Language Models31 Aug
  9. 2023Position-Enhanced Visual Instruction Tuning for Multimodal Large Language Models25 Aug
  10. 2023VIGC: Visual Instruction Generation and Correction24 Aug
  11. 2023StableLLaVA: Enhanced Visual Instruction Tuning with Synthesized Image-Dialogue Data20 Aug
  12. 2023TinyLVLM-eHub: Towards Comprehensive and Efficient Evaluation for Large Vision-Language Models7 Aug
  13. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities4 Aug
  14. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehension30 Jul
  15. 2023ChatSpot: Bootstrapping Multimodal LLMs via Precise Referring Instruction Tuning18 Jul
  16. 2023BuboGPT: Enabling Visual Grounding in Multi-Modal LLMs17 Jul
  17. 2023MMBench: Is Your Multi-modal Model an All-around Player?12 Jul
  18. 2023Visual Instruction Tuning with Polite Flamingo3 Jul
  19. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magic27 Jun
  20. 2023A Survey on Multimodal Large Language Models23 Jun
  21. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models23 Jun
  22. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning11 May
  23. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuning5 May

Other references · 116

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2023Segment Anythingcited by 8,158
  5. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  6. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
  7. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,290
  8. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  9. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  10. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  11. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,274
  12. 2023Qwen Technical Reportcited by 3,273
38 more
  1. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  2. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  3. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  4. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  5. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
  6. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  7. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  8. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,813
  9. 2023Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflectioncited by 1,800
  10. 2024Mixtral of Expertscited by 1,785
  11. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
  12. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,582
  13. 2022DINO: DETR with Improved DeNoising Anchor Boxes for End-to-End Object Detectioncited by 1,306
  14. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
  15. 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Daycited by 1,287
  16. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,075
  17. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,047
  18. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
  19. 2022Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Taskscited by 858
  20. 2023VideoChat: Chat-Centric Video Understandingcited by 851
  21. 2023Gemini: A Family of Highly Capable Multimodal Modelscited by 810
  22. 2023LISA: Reasoning Segmentation via Large Language Modelcited by 770
  23. 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 704
  24. 2023Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Modelscited by 659
  25. 2023MiniGPT-v2: large language model as a unified interface for vision-language multi-task learningcited by 617
  26. 2023LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Modelcited by 602
  27. 2023OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Modelscited by 560
  28. 2023Language Is Not All You Need: Aligning Perception with Language Modelscited by 543
  29. 2023LLaMA-VID: An Image is Worth 2 Tokens in Large Language Modelscited by 539
  30. 2023MM-REACT: Prompting ChatGPT for Multimodal Reasoning and Actioncited by 511
  31. 2023VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Taskscited by 499
  32. 2023Ferret: Refer and Ground Anything Anywhere at Any Granularitycited by 498
  33. 2023DriveGPT4: Interpretable End-to-end Autonomous Driving via Large Language Modelcited by 455
  34. 2023GLaMM: Pixel Grounding Large Multimodal Modelcited by 447
  35. 2023Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuningcited by 439
  36. 2023TimeChat: A Time-sensitive Multimodal Large Language Model for Long Video Understandingcited by 399
  37. 2023PMC-VQA: Visual Instruction Tuning for Medical Visual Question Answeringcited by 333
  38. 2023PandaGPT: One Model To Instruction-Follow Them Allcited by 330

50 most cited of 116 All 116 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 15; 2025: 75; 2026: 44.

134 papers.

Sort:

By area:

  1. 2024Divide, Conquer and Combine: A Training-Free Framework for High-Resolution Image Perception in Multimodal Large Language Modelscited by 151
  2. 2025MMSU: A Massive Multi-task Spoken Language Understanding and Reasoning Benchmarkcited by 101
  3. 2024Wiki-LLaVA: Hierarchical Retrieval-Augmented Generation for Multimodal LLMscited by 79
  4. 2025Don't Just Chase "Highlighted Tokens" in MLLMs: Revisiting Visual Holistic Context Retentioncited by 45
  5. 2025A Frustratingly Simple Yet Highly Effective Attack Baseline: Over 90% Success Rate Against the Strong Black-box Models of GPT-4.5/4o/o1cited by 34
  6. 2025Seeing the Arrow of Time in Large Multimodal Modelscited by 28
  7. 2024Next Token Prediction Towards Multimodal Intelligence: A Comprehensive Surveycited by 27
  8. 2025LLaVA-MORE: A Comparative Study of LLMs and Visual Backbones for Enhanced Visual Instruction Tuningcited by 27
  9. 2025VERIFY: A Benchmark of Visual Explanation and Reasoning for Investigating Multimodal Reasoning Fidelitycited by 23
  10. 2024LLM-based Multi-Agent Systems: Techniques and Business Perspectivescited by 22
  11. 2024OBI-Bench: Can LMMs Aid in Study of Ancient Script on Oracle Bones?cited by 22
  12. 2024From Specific-MLLMs to Omni-MLLMs: A Survey on MLLMs Aligned with Multi-modalitiescited by 22
38 more
  1. 2025Reducing Object Hallucination in Large Audio-Language Models via Audio-Aware Decodingcited by 22
  2. 2024The Evolution of Multimodal Model Architecturescited by 21
  3. 2025Hyperbolic Deep Learning for Foundation Models: A Surveycited by 20
  4. 2025DocPruner: A Storage-Efficient Framework for Multi-Vector Visual Document Retrieval via Adaptive Patch-Level Embedding Pruningcited by 19
  5. 2025AirCopBench: A Benchmark for Multi-drone Collaborative Embodied Perception and Reasoningcited by 19
  6. 2025Hybrid Token Compression for Vision-Language Modelscited by 19
  7. 2025Task-Oriented Feature Compression for Multimodal Understanding via Device-Edge Co-Inferencecited by 18
  8. 2025Benchmarking Multimodal Knowledge Conflict for Large Multimodal Modelscited by 18
  9. 2025MME-VideoOCR: Evaluating OCR-Based Capabilities of Multimodal LLMs in Video Scenarioscited by 18
  10. 2025MGA: Memory-Driven GUI Agent for Observation-Centric Interactioncited by 18
  11. 2025Recitation over Reasoning: How Cutting-Edge Language Models Can Fail on Elementary School-Level Reasoning Problems?cited by 15
  12. 2024The Synergy between Data and Multi-Modal Large Language Models: A Survey from Co-Development Perspectivecited by 14
  13. 2026MMOU: A Massive Multi-Task Omni Understanding and Reasoning Benchmark for Long and Complex Real-World Videoscited by 14
  14. 2024BRIDGE: Bridging Gaps in Image Captioning Evaluation with Stronger Visual Cuescited by 13
  15. 2025Recurrence-Enhanced Vision-and-Language Transformers for Robust Multimodal Document Retrievalcited by 13
  16. 2025GM-PRM: A Generative Multimodal Process Reward Model for Multimodal Mathematical Reasoningcited by 13
  17. 2025Texture or Semantics? Vision-Language Models Get Lost in Font Recognitioncited by 11
  18. 2025Hallucination at a Glance: Controlled Visual Edits and Fine-Grained Multimodal Learningcited by 11
  19. 2025ViPER: Empowering the Self-Evolution of Visual Perception Abilities in Vision-Language Modelcited by 11
  20. 2025Can Visual Input Be Compressed? A Visual Token Compression Benchmark for Large Multimodal Modelscited by 11
  21. 2024Data-Juicer Sandbox: A Feedback-Driven Suite for Multimodal Data-Model Co-developmentcited by 10
  22. 2025TUNA: Comprehensive Fine-grained Temporal Understanding Evaluation on Dense Dynamic Videoscited by 9
  23. 2025Towards Understanding Visual Grounding in Visual Language Modelscited by 9
  24. 2025VideoMiner: Iteratively Grounding Key Frames of Hour-Long Videos via Tree-based Group Relative Policy Optimizationcited by 9
  25. 2026Do Audio-Visual Large Language Models Really See and Hear?cited by 9
  26. 2024Multi-modal Generative AI: Multi-modal LLMs, Diffusions, and the Unificationcited by 8
  27. 2025PhysicsArena: The First Multimodal Physics Reasoning Benchmark Exploring Variable, Process, and Solution Dimensionscited by 8
  28. 2025A Survey on MLLM-based Visually Rich Document Understanding: Methods, Challenges, and Emerging Trendscited by 8
  29. 2025Multimodal LLM-assisted Evolutionary Search for Programmatic Control Policiescited by 8
  30. 2025Seeing is Understanding: Unlocking Causal Attention into Modality-Mutual Attention for Multimodal LLMscited by 7
  31. 2025VisuRiddles: Fine-grained Perception is a Primary Bottleneck for Multimodal Large Language Models in Abstract Visual Reasoningcited by 7
  32. 2025MoMA: A Mixture-of-Multimodal-Agents Architecture for Enhancing Clinical Prediction Modellingcited by 7
  33. 2025Mitigating Hallucinations in Multimodal LLMs via Object-aware Preference Optimizationcited by 7
  34. 2025MOSS-ChatV: Reinforcement Learning with Process Reasoning Reward for Video Temporal Reasoningcited by 7
  35. 2025ORIGAMISPACE: Benchmarking Multimodal LLMs in Multi-Step Spatial Reasoning with Mathematical Constraintscited by 7
  36. 2024MedMax: Mixed-Modal Instruction Tuning for Training Biomedical Assistantscited by 6
  37. 2025R-Genie: Reasoning-Guided Generative Image Editingcited by 6
  38. 2025Higher Satisfaction, Lower Cost: A Technical Report on How LLMs Revolutionize Meituan's Intelligent Interaction Systemscited by 6

50 most cited of 134 All 134 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024A Comprehensive Survey and Guide to Multimodal Large Language Models in Vision-Language Tasks
  2. 2024A Comprehensive Review of Multimodal Large Language Models: Performance and Challenges Across Different Tasks
  3. 2024A Survey of Multimodal Large Language Model from A Data-centric Perspective
  4. 2024Efficient Multimodal Large Language Models: A Survey
  5. 2023SEED-Bench-2: Benchmarking Multimodal Large Language Models
  6. 2025Multimodal Large Language Models for Text-rich Image Understanding: A Comprehensive Review
  7. 2024A Survey on Benchmarks of Multimodal Large Language Models
  8. 2024Explainable and Interpretable Multimodal Large Language Models: A Comprehensive Survey