mlsanityData to 7 Oct 2026

Efficient Multimodal Large Language Models: A Survey

arXiv 2405.10739 ↗First posted 17 May 2024cs.CV cs.AIcited by 95

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

In the past year, Multimodal Large Language Models (MLLMs) have demonstrated remarkable performance in tasks such as visual question answering, visual understanding and reasoning. However, the extensive model size and high training and inference costs have hindered the widespread application of MLLMs in academia and industry. Thus, studying efficient and lightweight MLLMs has enormous potential, especially in edge computing scenarios. In this survey, we provide a comprehensive and systematic review of the current state of efficient MLLMs. Specifically, we summarize the timeline of representative efficient MLLMs, research state of efficient structures and strategies, and the applications. Finally, we discuss the limitations of current efficient MLLM research and promising future directions. Please refer to our GitHub repository for more details: https://github.com/lijiannuist/Efficient-Multimodal-LLMs-Survey.

Key facts. Cited by 95 papers in the map (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 180 of its 227 references that are in the map. First posted on arXiv 17 May 2024.

Builds on

180 of 227 references.

Closest ancestors · 28

Same direction, newest first

  1. 2024MoVA: Adapting Mixture of Vision Experts to Multimodal Context19 Apr
  2. 2024TextHawk: Exploring Efficient Fine-Grained Perception of Multimodal Large Language Models14 Apr
  3. 2024InternLM-XComposer2-4KHD: A Pioneering Large Vision-Language Model Handling Resolutions from 336 Pixels to 4K HD9 Apr
  4. 2024Cobra: Extending Mamba to Multi-Modal Large Language Model for Efficient Inference21 Mar
  5. 2024VL-Mamba: Exploring State Space Models for Multimodal Learning20 Mar
  6. 2024HyperLLaVA: Dynamic Visual and Language Expert Tuning for Multimodal Large Language Models20 Mar
  7. 2024MM1: Methods, Analysis & Insights from Multimodal LLM Pre-training14 Mar
  8. 2024TinyLLaVA: A Framework of Small-scale Large Multimodal Models22 Feb
  9. 2024Efficient Multimodal Learning from Data-centric Perspective18 Feb
  10. 2024SPHINX-X: Scaling Data and Parameters for a Family of Multi-modal Large Language Models8 Feb
  11. 2024MoE-LLaVA: Mixture of Experts for Large Vision-Language Models29 Jan
  12. 2024MM-LLMs: Recent Advances in MultiModal Large Language Models24 Jan
16 more
  1. 2023TinyGPT-V: Efficient Multimodal Large Language Model via Small Backbones28 Dec
  2. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks21 Dec
  3. 2023Generative Multimodal Models are In-Context Learners20 Dec
  4. 2023VILA: On Pre-training for Visual Language Models12 Dec
  5. 2023Honeybee: Locality-enhanced Projector for Multimodal LLM11 Dec
  6. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captions21 Nov
  7. 2023Monkey: Image Resolution and Text Label Are Important Things for Large Multi-modal Models11 Nov
  8. 2023mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaboration7 Nov
  9. 2023Improved Baselines with Visual Instruction Tuning5 Oct
  10. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities4 Aug
  11. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehension30 Jul
  12. 2023MMBench: Is Your Multi-modal Model an All-around Player?12 Jul
  13. 2023A Survey on Multimodal Large Language Models23 Jun
  14. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models23 Jun
  15. 2023OBELICS: An Open Web-Scale Filtered Dataset of Interleaved Image-Text Documents21 Jun
  16. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning11 May

Other references · 152

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  5. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  6. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  7. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  8. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  9. 2023Segment Anythingcited by 8,158
  10. 2023Visual Instruction Tuningcited by 7,792
  11. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  12. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
38 more
  1. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
  2. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  3. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  4. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  5. 2022A ConvNet for the 2020scited by 3,545
  6. 2023Qwen Technical Reportcited by 3,273
  7. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  8. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  9. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  10. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  11. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  12. 2021Efficiently Modeling Long Sequences with Structured State Spacescited by 2,424
  13. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  14. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,883
  15. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  16. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
  17. 2019Towards VQA Models That Can Readcited by 1,696
  18. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
  19. 2020Reformer: The Efficient Transformercited by 1,658
  20. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  21. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,582
  22. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
  23. 2020Linformer: Self-Attention with Linear Complexitycited by 1,462
  24. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,461
  25. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,449
  26. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
  27. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,385
  28. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,311
  29. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,288
  30. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,234
  31. 2019LVIS: A Dataset for Large Vocabulary Instance Segmentationcited by 1,215
  32. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
  33. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,114
  34. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
  35. 2015Generation and Comprehension of Unambiguous Object Descriptionscited by 897
  36. 2022Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learningcited by 886
  37. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 852
  38. 2023VideoChat: Chat-Centric Video Understandingcited by 851

50 most cited of 152 All 152 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 33; 2025: 33; 2026: 29.

95 papers.

Sort:

By area:

  1. 2024MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmarkcited by 494
  2. 2024SpatialBot: Precise Spatial Understanding with Vision Language Modelscited by 146
  3. 2024WavChat: A Survey of Spoken Dialogue Modelscited by 88
  4. 2024A Survey on Benchmarks of Multimodal Large Language Modelscited by 81
  5. 2024MM1.5: Methods, Analysis & Insights from Multimodal LLM Fine-tuningcited by 75
  6. 2024Prompt Compression for Large Language Models: A Surveycited by 65
  7. 2024Dynamic-LLaVA: Efficient Multimodal Large Language Models via Dynamic Vision-language Context Sparsificationcited by 59
  8. 2024Explainable and Interpretable Multimodal Large Language Models: A Comprehensive Surveycited by 52
  9. 2024FrameFusion: Combining Similarity and Importance for Video Token Reduction on Large Vision Language Modelscited by 52
  10. 2024MMAD: A Comprehensive Benchmark for Multimodal Large Language Models in Industrial Anomaly Detectioncited by 46
  11. 2024Mitigating Modality Prior-Induced Hallucinations in Multimodal Large Language Models via Deciphering Attention Causalitycited by 42
  12. 2025StreamingTOM: Streaming Token Compression for Efficient Video Understandingcited by 41
38 more
  1. 2025Harnessing the Reasoning Economy: A Survey of Efficient Reasoning for Large Language Modelscited by 35
  2. 2024Unveiling the Ignorance of MLLMs: Seeing Clearly, Answering Incorrectlycited by 29
  3. 2025Vision-Language Models for Edge Networks: A Comprehensive Surveycited by 29
  4. 2024Next Token Prediction Towards Multimodal Intelligence: A Comprehensive Surveycited by 27
  5. 2025Agentic Plan Caching: Test-Time Memory for Fast and Cost-Efficient LLM Agentscited by 23
  6. 2024Beyond Logit Lens: Contextual Embeddings for Robust Hallucination Detection & Grounding in VLMscited by 22
  7. 2024From Specific-MLLMs to Omni-MLLMs: A Survey on MLLMs Aligned with Multi-modalitiescited by 22
  8. 2024Imp: Highly Capable Large Multimodal Models for Mobile Devicescited by 19
  9. 2025A Survey on Efficient Vision-Language Modelscited by 19
  10. 2026VisionTrim: Unified Vision Token Compression for Training-Free MLLM Accelerationcited by 16
  11. 2025FC-Attack: Jailbreaking Multimodal Large Language Models via Auto-Generated Flowchartscited by 15
  12. 2024The Synergy between Data and Multi-Modal Large Language Models: A Survey from Co-Development Perspectivecited by 14
  13. 2024From Efficient Multimodal Models to World Models: A Surveycited by 13
  14. 2024γ-MoD: Exploring Mixture-of-Depth Adaptation for Multimodal Large Language Modelscited by 13
  15. 2024Promoting cross-modal representations to improve multimodal foundation models for physiological signalscited by 13
  16. 2024WalkVLM:Aid Visually Impaired People Walking by Vision Language Modelcited by 13
  17. 2026Explore with Long-term Memory: A Benchmark and Multimodal LLM-based Reinforcement Learning Framework for Embodied Explorationcited by 13
  18. 2025When Semantics Mislead Vision: Mitigating Large Multimodal Models Hallucinations in Scene Text Spotting and Understandingcited by 12
  19. 2025Skip-Vision: Efficient and Scalable Acceleration of Vision-Language Models via Adaptive Token Skippingcited by 11
  20. 2025Poison as Cure: Visual Noise for Mitigating Object Hallucinations in LVMscited by 9
  21. 2024Multi-modal Generative AI: Multi-modal LLMs, Diffusions, and the Unificationcited by 8
  22. 2024Cost-effective Instruction Learning for Pathology Vision and Language Analysiscited by 7
  23. 2024LLaVA-VSD: Large Language-and-Vision Assistant for Visual Spatial Descriptioncited by 7
  24. 2024FedMLLM: Federated Fine-tuning MLLM on Multimodal Heterogeneity Datacited by 7
  25. 2024freePruner: A Training-free Approach for Large Multimodal Model Accelerationcited by 7
  26. 2025LightVLM: Acceleraing Large Multimodal Models with Pyramid Token Merging and KV Cache Compressioncited by 7
  27. 2025CMIE: Combining MLLM Insights with External Evidence for Explainable Out-of-Context Misinformation Detectioncited by 6
  28. 2026Dynamic Multimodal Activation Steering for Hallucination Mitigation in Large Vision-Language Modelscited by 6
  29. 2025Task-Oriented Semantic Communication in Large Multimodal Models-based Vehicle Networkscited by 5
  30. 2026Gated Relational Alignment via Confidence-based Distillation for Efficient VLMscited by 5
  31. 2024PAR: Prompt-Aware Token Reduction Method for Efficient Large Multimodal Modelscited by 4
  32. 2024Self-adaptive Multimodal Retrieval-Augmented Generationcited by 4
  33. 2025Retrieval-augmented in-context learning for multimodal large language models in disease classificationcited by 4
  34. 2025EgoPrune: Efficient Token Pruning for Egomotion Video Reasoning in Embodied Agentcited by 4
  35. 2025HUMORCHAIN: Theory-Guided Multi-Stage Reasoning for Interpretable Multimodal Humor Generationcited by 4
  36. 2026EvoComp: Learning Visual Token Compression for Multimodal Large Language Models via Semantic-Guided Evolutionary Labelingcited by 4
  37. 2024Less is More: A Simple yet Effective Token Reduction Method for Efficient Multi-modal LLMscited by 3
  38. 2025Revisiting Funnel Transformers for Modern LLM Architectures with Comprehensive Ablations in Training and Inference Configurationscited by 3

50 most cited of 95 All 95 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024The Revolution of Multimodal Large Language Models: A Survey
  2. 2024A Comprehensive Review of Multimodal Large Language Models: Performance and Challenges Across Different Tasks
  3. 2024A Survey of Multimodal Large Language Model from A Data-centric Perspective
  4. 2023SEED-Bench-2: Benchmarking Multimodal Large Language Models
  5. 2025Multimodal Large Language Models for Text-rich Image Understanding: A Comprehensive Review
  6. 2024A Comprehensive Survey and Guide to Multimodal Large Language Models in Vision-Language Tasks
  7. 2025Benchmarking Large and Small MLLMs
  8. 2024EE-MLLM: A Data-Efficient and Compute-Efficient Multimodal Large Language Model