mlsanityData to 7 Oct 2026

MM-LLMs: Recent Advances in MultiModal Large Language Models

arXiv 2401.13601 ↗First posted 24 Jan 2024cs.CLcited by 277

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

In the past year, MultiModal Large Language Models (MM-LLMs) have undergone substantial advancements, augmenting off-the-shelf LLMs to support MM inputs or outputs via cost-effective training strategies. The resulting models not only preserve the inherent reasoning and decision-making capabilities of LLMs but also empower a diverse range of MM tasks. In this paper, we provide a comprehensive survey aimed at facilitating further research of MM-LLMs. Initially, we outline general design formulations for model architecture and training pipeline. Subsequently, we introduce a taxonomy encompassing 126 MM-LLMs, each characterized by its specific formulations. Furthermore, we review the performance of selected MM-LLMs on mainstream benchmarks and summarize key training recipes to enhance the potency of MM-LLMs. Finally, we explore promising directions for MM-LLMs while concurrently maintaining a real-time tracking website for the latest developments in the field. We hope that this survey contributes to the ongoing advancement of the MM-LLMs domain.

Key facts. Cited by 277 papers in the map (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 184 of its 275 references that are in the map. First posted on arXiv 24 Jan 2024.

Builds on

184 of 275 references.

Closest ancestors · 25

Same direction, newest first

  1. 2023TinyGPT-V: Efficient Multimodal Large Language Model via Small Backbones28 Dec
  2. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks21 Dec
  3. 2023Generative Multimodal Models are In-Context Learners20 Dec
  4. 2023Osprey: Pixel Understanding with Visual Instruction Tuning15 Dec
  5. 2023BenchLMM: Benchmarking Cross-style Visual Capability of Large Multimodal Models5 Dec
  6. 2023CoDi-2: In-Context, Interleaved, and Interactive Any-to-Any Generation30 Nov
  7. 2023Multimodal Large Language Models: A Survey22 Nov
  8. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captions21 Nov
  9. 2023Monkey: Image Resolution and Text Label Are Important Things for Large Multi-modal Models11 Nov
  10. 2023LLaVA-Plus: Learning to Use Tools for Creating Multimodal Agents9 Nov
  11. 2023mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaboration7 Nov
  12. 2023Improved Baselines with Visual Instruction Tuning5 Oct
13 more
  1. 2023Aligning Large Multimodal Models with Factually Augmented RLHF25 Sep
  2. 2023DreamLLM: Synergistic Multimodal Comprehension and Creation20 Sep
  3. 2023Scaling Autoregressive Multi-Modal Models: Pretraining and Instruction Tuning5 Sep
  4. 2023Large Multilingual Models Pivot Zero-Shot Multimodal Learning across Languages23 Aug
  5. 2023StableLLaVA: Enhanced Visual Instruction Tuning with Synthesized Image-Dialogue Data20 Aug
  6. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities4 Aug
  7. 2023ChatSpot: Bootstrapping Multimodal LLMs via Precise Referring Instruction Tuning18 Jul
  8. 2023BuboGPT: Enabling Visual Grounding in Multi-Modal LLMs17 Jul
  9. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models23 Jun
  10. 2023Generating Images with Multimodal Language Models26 May
  11. 2023SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilities18 May
  12. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning11 May
  13. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuning5 May

Other references · 159

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  5. 2020Language Models are Few-Shot Learnerscited by 32,199
  6. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  7. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  8. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  9. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  10. 2022Training language models to follow instructions with human feedbackcited by 15,712
  11. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  12. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
38 more
  1. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  2. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  3. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  4. 2023Segment Anythingcited by 8,158
  5. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  6. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  7. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  8. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  9. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  10. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  11. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  12. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  13. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  14. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  15. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  16. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  17. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  18. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  19. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  20. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  21. 2023Retrieval-Augmented Generation for Large Language Models: A Surveycited by 2,185
  22. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  23. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  24. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  25. 2019Towards VQA Models That Can Readcited by 1,696
  26. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
  27. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,582
  28. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
  29. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,461
  30. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
  31. 2022DINO: DETR with Improved DeNoising Anchor Boxes for End-to-End Object Detectioncited by 1,306
  32. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,288
  33. 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Daycited by 1,287
  34. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
  35. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
  36. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,175
  37. 2022Reproducible scaling laws for contrastive language-image learningcited by 1,163
  38. 2023ImageBind: One Embedding Space To Bind Them Allcited by 1,162

50 most cited of 159 All 159 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 98; 2025: 130; 2026: 49.

277 papers.

Sort:

By area:

  1. 2024LLaVA-PruMerge: Adaptive Token Reduction for Efficient Large Multimodal Modelscited by 394
  2. 2025From System 1 to System 2: A Survey of Reasoning Large Language Modelscited by 319
  3. 2024mPLUG-DocOwl 1.5: Unified Structure Learning for OCR-free Document Understandingcited by 220
  4. 2024MobileVLM V2: Faster and Stronger Baseline for Vision Language Modelcited by 172
  5. 2024TokenPacker: Efficient Visual Projector for Multimodal LLMcited by 170
  6. 2024LLM Inference Unveiled: Survey and Roofline Model Insightscited by 147
  7. 2024A Survey on Large Language Model Acceleration based on KV Cache Managementcited by 125
  8. 2025On The Landscape of Spoken Language Models: A Comprehensive Surveycited by 116
  9. 2024AdaShield: Safeguarding Multimodal Large Language Models from Structure-based Attack via Adaptive Shield Promptingcited by 113
  10. 2024Efficient Multimodal Large Language Models: A Surveycited by 95
  11. 2024A Survey on Multilingual Large Language Models: Corpora, Alignment, and Biascited by 82
  12. 2024Multimodal Alignment and Fusion: A Surveycited by 73
38 more
  1. 2024BRAVE: Broadening the visual encoding of vision-language modelscited by 68
  2. 2024Exploring the Frontier of Vision-Language Models: A Survey of Current Methodologies and Future Directionscited by 66
  3. 2024Pangea: A Fully Open Multilingual Multimodal LLM for 39 Languagescited by 63
  4. 2025SpatialCoT: Advancing Spatial Reasoning through Coordinate Alignment and Chain-of-Thought for Embodied Task Planningcited by 62
  5. 2024Visual-RolePlay: Universal Jailbreak Attack on MultiModal Large Language Models via Role-playing Image Charactercited by 61
  6. 2025Large language models for automated scholarly paper review: A surveycited by 59
  7. 2024A Comprehensive Review of Multimodal Large Language Models: Performance and Challenges Across Different Taskscited by 57
  8. 2024KG-RAG: Bridging the Gap Between Knowledge and Creativitycited by 55
  9. 2024What Makes Multimodal In-Context Learning Work?cited by 54
  10. 2024Explainable and Interpretable Multimodal Large Language Models: A Comprehensive Surveycited by 52
  11. 2025VisualPuzzles: Decoupling Multimodal Reasoning Evaluation from Domain Knowledgecited by 52
  12. 2025PACT: Pruning and Clustering-Based Token Reduction for Faster Visual Language Modelscited by 49
  13. 2025T2ISafety: Benchmark for Assessing Fairness, Toxicity, and Privacy in Image Generationcited by 47
  14. 2024A Comprehensive Survey of Large Language Models and Multimodal Large Language Models in Medicinecited by 45
  15. 2024Single Image Unlearning: Efficient Machine Unlearning in Multimodal Large Language Modelscited by 44
  16. 2024Mitigating Modality Prior-Induced Hallucinations in Multimodal Large Language Models via Deciphering Attention Causalitycited by 42
  17. 2024Graphic Design with Large Multimodal Modelcited by 38
  18. 2025Parameters vs. Context: Fine-Grained Control of Knowledge Reliance in Language Modelscited by 38
  19. 2025UFO2: The Desktop AgentOScited by 38
  20. 2024SOLO: A Single Transformer for Scalable Vision-Language Modelingcited by 35
  21. 2025Visual Large Language Models for Generalized and Specialized Applicationscited by 35
  22. 2025A Survey of Direct Preference Optimizationcited by 35
  23. 2025CausalVQA: A Physically Grounded Causal Reasoning Benchmark for Video Modelscited by 35
  24. 2025OS Agents: A Survey on MLLM-based Agents for General Computing Devices Usecited by 35
  25. 2025A Frustratingly Simple Yet Highly Effective Attack Baseline: Over 90% Success Rate Against the Strong Black-box Models of GPT-4.5/4o/o1cited by 34
  26. 2024How to Continually Adapt Text-to-Image Diffusion Models for Flexible Customization?cited by 32
  27. 2024GITA: Graph to Visual and Textual Integration for Vision-Language Graph Reasoningcited by 31
  28. 2024Probing the limitations of multimodal language models for chemistry and materials researchcited by 31
  29. 2024What is the Visual Cognition Gap between Humans and Multimodal LLMs?cited by 30
  30. 2025Fine-Grained Preference Optimization Improves Spatial Reasoning in VLMscited by 30
  31. 2024Unveiling the Ignorance of MLLMs: Seeing Clearly, Answering Incorrectlycited by 29
  32. 2024BELM: Bidirectional Explicit Linear Multi-step Sampler for Exact Inversion in Diffusion Modelscited by 29
  33. 2025AutoReproduce: Automatic AI Experiment Reproduction with Paper Lineagecited by 29
  34. 2024Uncertainty-Aware Evaluation for Vision-Language Modelscited by 28
  35. 2024C-RAG: Certified Generation Risks for Retrieval-Augmented Language Modelscited by 27
  36. 2024EventGPT: Event Stream Understanding with Multimodal Large Language Modelscited by 27
  37. 2024Next Token Prediction Towards Multimodal Intelligence: A Comprehensive Surveycited by 27
  38. 2024Facial Affective Behavior Analysis with Instruction Tuningcited by 26

50 most cited of 277 All 277 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023A Comprehensive Overview of Large Language Models
  2. 2025Self-Improvement in Multimodal Large Language Models: A Survey
  3. 2024Survey of different Large Language Model Architectures: Trends, Benchmarks, and Challenges
  4. 2024A Review of Multi-Modal Large Language and Vision Models
  5. 2024MME-Survey: A Comprehensive Survey on Evaluation of Multimodal LLMs
  6. 2025Domain Specific Benchmarks for Evaluating Multimodal Large Language Models
  7. 2024A critical review of methods and challenges in large language models
  8. 2025Empowering Multimodal LLMs with External Tools: A Comprehensive Survey