# Efficient Multimodal Large Language Models: A Survey

arXiv: https://arxiv.org/abs/2405.10739 · HTML: https://mlsanity.com/paper/2405.10739/

## Key facts

- Cited by 96 papers in the map (data through 14 Oct 2026).
- Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 180 of its 227 references that are in the map.
- First posted on arXiv 17 May 2024.

## Placed in

- Multimodal LLMs / vision-language models / [MLLM training and visual instruction tuning](https://mlsanity.com/mllm/mllm-training-and-visual-instruction-tuning/)

## Builds on (most cited first, 30 of 180)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [Mamba: Linear-Time Sequence Modeling with Selective State Spaces](https://mlsanity.com/paper/2312.00752/) (2023)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [Improved Baselines with Visual Instruction Tuning](https://mlsanity.com/paper/2310.03744/) (2023)
- [Flamingo: a Visual Language Model for Few-Shot Learning](https://mlsanity.com/paper/2204.14198/) (2022)
- [LAION-5B: An open large-scale dataset for training next generation image-text models](https://mlsanity.com/paper/2210.08402/) (2022)
- [A ConvNet for the 2020s](https://mlsanity.com/paper/2201.03545/) (2022)
- [Qwen Technical Report](https://mlsanity.com/paper/2309.16609/) (2023)
- [Sigmoid Loss for Language Image Pre-Training](https://mlsanity.com/paper/2303.15343/) (2023)
- [Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity](https://mlsanity.com/paper/2101.03961/) (2021)
- [InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning](https://mlsanity.com/paper/2305.06500/) (2023)
- [MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models](https://mlsanity.com/paper/2304.10592/) (2023)
- [Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations](https://mlsanity.com/paper/1602.07332/) (2016)
- [InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks](https://mlsanity.com/paper/2312.14238/) (2023)
- [Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering](https://mlsanity.com/paper/1612.00837/) (2016)
- [Efficiently Modeling Long Sequences with Structured State Spaces](https://mlsanity.com/paper/2111.00396/) (2021)
- [MMBench: Is Your Multi-modal Model an All-around Player?](https://mlsanity.com/paper/2307.06281/) (2023)
- [MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI](https://mlsanity.com/paper/2311.16502/) (2023)
- [Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering](https://mlsanity.com/paper/2209.09513/) (2022)

## Built on by (most cited first, 30 of 96)

- [MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmark](https://mlsanity.com/paper/2409.02813/) (2024)
- [SpatialBot: Precise Spatial Understanding with Vision Language Models](https://mlsanity.com/paper/2406.13642/) (2024)
- [WavChat: A Survey of Spoken Dialogue Models](https://mlsanity.com/paper/2411.13577/) (2024)
- [A Survey on Benchmarks of Multimodal Large Language Models](https://mlsanity.com/paper/2408.08632/) (2024)
- [MM1.5: Methods, Analysis & Insights from Multimodal LLM Fine-tuning](https://mlsanity.com/paper/2409.20566/) (2024)
- [Prompt Compression for Large Language Models: A Survey](https://mlsanity.com/paper/2410.12388/) (2024)
- [Dynamic-LLaVA: Efficient Multimodal Large Language Models via Dynamic Vision-language Context Sparsification](https://mlsanity.com/paper/2412.00876/) (2024)
- [FrameFusion: Combining Similarity and Importance for Video Token Reduction on Large Vision Language Models](https://mlsanity.com/paper/2501.01986/) (2024)
- [Explainable and Interpretable Multimodal Large Language Models: A Comprehensive Survey](https://mlsanity.com/paper/2412.02104/) (2024)
- [MMAD: A Comprehensive Benchmark for Multimodal Large Language Models in Industrial Anomaly Detection](https://mlsanity.com/paper/2410.09453/) (2024)
- [StreamingTOM: Streaming Token Compression for Efficient Video Understanding](https://mlsanity.com/paper/2510.18269/) (2025)
- [Mitigating Modality Prior-Induced Hallucinations in Multimodal Large Language Models via Deciphering Attention Causality](https://mlsanity.com/paper/2410.04780/) (2024)
- [Harnessing the Reasoning Economy: A Survey of Efficient Reasoning for Large Language Models](https://mlsanity.com/paper/2503.24377/) (2025)
- [Unveiling the Ignorance of MLLMs: Seeing Clearly, Answering Incorrectly](https://mlsanity.com/paper/2406.10638/) (2024)
- [Vision-Language Models for Edge Networks: A Comprehensive Survey](https://mlsanity.com/paper/2502.07855/) (2025)
- [Next Token Prediction Towards Multimodal Intelligence: A Comprehensive Survey](https://mlsanity.com/paper/2412.18619/) (2024)
- [Agentic Plan Caching: Test-Time Memory for Fast and Cost-Efficient LLM Agents](https://mlsanity.com/paper/2506.14852/) (2025)
- [Beyond Logit Lens: Contextual Embeddings for Robust Hallucination Detection & Grounding in VLMs](https://mlsanity.com/paper/2411.19187/) (2024)
- [From Specific-MLLMs to Omni-MLLMs: A Survey on MLLMs Aligned with Multi-modalities](https://mlsanity.com/paper/2412.11694/) (2024)
- [Imp: Highly Capable Large Multimodal Models for Mobile Devices](https://mlsanity.com/paper/2405.12107/) (2024)
- [A Survey on Efficient Vision-Language Models](https://mlsanity.com/paper/2504.09724/) (2025)
- [FC-Attack: Jailbreaking Multimodal Large Language Models via Auto-Generated Flowcharts](https://mlsanity.com/paper/2502.21059/) (2025)
- [VisionTrim: Unified Vision Token Compression for Training-Free MLLM Acceleration](https://mlsanity.com/paper/2601.22674/) (2026)
- [The Synergy between Data and Multi-Modal Large Language Models: A Survey from Co-Development Perspective](https://mlsanity.com/paper/2407.08583/) (2024)
- [γ-MoD: Exploring Mixture-of-Depth Adaptation for Multimodal Large Language Models](https://mlsanity.com/paper/2410.13859/) (2024)
- [From Efficient Multimodal Models to World Models: A Survey](https://mlsanity.com/paper/2407.00118/) (2024)
- [Promoting cross-modal representations to improve multimodal foundation models for physiological signals](https://mlsanity.com/paper/2410.16424/) (2024)
- [WalkVLM:Aid Visually Impaired People Walking by Vision Language Model](https://mlsanity.com/paper/2412.20903/) (2024)
- [Explore with Long-term Memory: A Benchmark and Multimodal LLM-based Reinforcement Learning Framework for Embodied Exploration](https://mlsanity.com/paper/2601.10744/) (2026)
- [Skip-Vision: Efficient and Scalable Acceleration of Vision-Language Models via Adaptive Token Skipping](https://mlsanity.com/paper/2503.21817/) (2025)

## Abstract (arXiv)

In the past year, Multimodal Large Language Models (MLLMs) have demonstrated remarkable performance in tasks such as visual question answering, visual understanding and reasoning. However, the extensive model size and high training and inference costs have hindered the widespread application of MLLMs in academia and industry. Thus, studying efficient and lightweight MLLMs has enormous potential, especially in edge computing scenarios. In this survey, we provide a comprehensive and systematic review of the current state of efficient MLLMs. Specifically, we summarize the timeline of representative efficient MLLMs, research state of efficient structures and strategies, and the applications. Finally, we discuss the limitations of current efficient MLLM research and promising future directions. Please refer to our GitHub repository for more details: https://github.com/lijiannuist/Efficient-Multimodal-LLMs-Survey.