mlsanityData to 14 Oct 2026

LLaVA-Mini: Efficient Image and Video Large Multimodal Models with One Vision Token

arXiv 2501.03895 ↗First posted 7 Jan 2025cs.CV cs.AI cs.CLcited by 166

On the map: Multimodal LLMs / vision-language models → Visual token pruning and efficiency

Abstract (arXiv)

The advent of real-time large multimodal models (LMMs) like GPT-4o has sparked considerable interest in efficient LMMs. LMM frameworks typically encode visual inputs into vision tokens (continuous representations) and integrate them and textual instructions into the context of large language models (LLMs), where large-scale parameters and numerous context tokens (predominantly vision tokens) result in substantial computational overhead. Previous efforts towards efficient LMMs always focus on replacing the LLM backbone with smaller models, while neglecting the crucial issue of token quantity. In this paper, we introduce LLaVA-Mini, an efficient LMM with minimal vision tokens. To achieve a high compression ratio of vision tokens while preserving visual information, we first analyze how LMMs understand vision tokens and find that most vision tokens only play a crucial role in the early layers of LLM backbone, where they mainly fuse visual information into text tokens. Building on this finding, LLaVA-Mini introduces modality pre-fusion to fuse visual information into text tokens in advance, thereby facilitating the extreme compression of vision tokens fed to LLM backbone into one token. LLaVA-Mini is a unified large multimodal model that can support the understanding of images, high-resolution images, and videos in an efficient manner. Experiments across 11 image-based and 7 video-based benchmarks demonstrate that LLaVA-Mini outperforms LLaVA-v1.5 with just 1 vision token instead of 576. Efficiency analyses reveal that LLaVA-Mini can reduce FLOPs by 77%, deliver low-latency responses within 40 milliseconds, and process over 10,000 frames of video on the GPU hardware with 24GB of memory.

Key facts. Cited by 166 papers in the map (data through 14 Oct 2026). Placed in Visual token pruning and efficiency, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 44 of its 59 references that are in the map. First posted on arXiv 7 Jan 2025.

Builds on

44 of 59 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024TokenPacker: Efficient Visual Projector for Multimodal LLM2 Jul
  2. 2024VoCo-LLaMA: Towards Vision Compression with Large Language Models18 Jun
  3. 2024LLaVA-PruMerge: Adaptive Token Reduction for Efficient Large Multimodal Models22 Mar
  4. 2024An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Models11 Mar

Other references · 40

  1. 2017Attention Is All You Needcited by 60,361
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2023Visual Instruction Tuningcited by 7,869
  5. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  6. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  7. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  8. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  9. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
  10. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  11. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,141
  12. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,919
28 more
  1. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,729
  2. 2019Towards VQA Models That Can Readcited by 1,718
  3. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,652
  4. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,471
  5. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,402
  6. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,239
  7. 2023MVBench: A Comprehensive Multi-modal Video Understanding Benchmarkcited by 1,109
  8. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,103
  9. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,062
  10. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,040
  11. 2022Token Merging: Your ViT But Fastercited by 1,021
  12. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 858
  13. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 781
  14. 2023EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understandingcited by 680
  15. 2023mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaborationcited by 547
  16. 2023LLaMA-VID: An Image is Worth 2 Tokens in Large Language Modelscited by 546
  17. 2023MovieChat: From Dense Token to Sparse Memory for Long Video Understandingcited by 542
  18. 2023TimeChat: A Time-sensitive Multimodal Large Language Model for Long Video Understandingcited by 406
  19. 2024mPLUG-Owl3: Towards Long Image-Sequence Understanding in Multi-Modal Large Language Modelscited by 291
  20. 2023SPHINX: The Joint Mixing of Weights, Tasks, and Visual Embeddings for Multi-modal Large Language Modelscited by 286
  21. 2024MA-LMM: Memory-Augmented Large Multimodal Model for Long-Term Video Understandingcited by 229
  22. 2024MobileVLM V2: Faster and Stronger Baseline for Vision Language Modelcited by 174
  23. 2024TinyLLaVA: A Framework of Small-scale Large Multimodal Modelscited by 164
  24. 2024LLM Inference Unveiled: Survey and Roofline Model Insightscited by 152
  25. 2023TinyGPT-V: Efficient Multimodal Large Language Model via Small Backbonescited by 77
  26. 2024ParGo: Bridging Vision-Language with Partial and Global Viewscited by 40
  27. 2024How Good is my Video LMM? Complex Video Reasoning and Robustness Evaluation Suite for Video-LMMscited by 30
  28. 2024DreamFrame: Enhancing Video Understanding via Automatically Generated QA and Style-Consistent Keyframescited by 13

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 90; 2026: 76.

166 papers.

Sort:

By area:

  1. 2025DeepEyes: Incentivizing "Thinking with Images" via Reinforcement Learningcited by 410
  2. 2025Thyme: Think Beyond Imagescited by 167
  3. 2025Magma: A Foundation Model for Multimodal AI Agentscited by 139
  4. 2025Beyond Attention or Similarity: Maximizing Conditional Diversity for Token Pruning in MLLMscited by 96
  5. 2025VideoRoPE: What Makes for Good Video Rotary Position Embedding?cited by 65
  6. 2025R1-Reward: Training Multimodal Reward Model Through Stable Reinforcement Learningcited by 65
  7. 2025TinyLLaVA-Video-R1: Towards Smaller LMMs for Video Reasoningcited by 62
  8. 2025StreamMem: Query-Agnostic KV Cache Memory for Streaming Video Understandingcited by 58
  9. 2026LLaVA-OneVision-2: Towards Next-Generation Perceptual Intelligencecited by 39
  10. 2025Scaling Video-Language Models to 10K Frames via Hierarchical Differential Distillationcited by 37
  11. 2025Vamba: Understanding Hour-Long Videos with Hybrid Mamba-Transformerscited by 36
  12. 2025The Better You Learn, The Smarter You Prune: Towards Efficient Vision-language-action Models via Differentiable Token Pruningcited by 36
38 more
  1. 2025When Large Vision-Language Model Meets Large Remote Sensing Imagery: Coarse-to-Fine Text-Guided Token Pruningcited by 35
  2. 2025Think Twice, Act Once: Token-Aware Compression and Action Reuse for Efficient Inference in Vision-Language-Action Modelscited by 30
  3. 2025A Survey of Token Compression for Efficient Multimodal Large Language Modelscited by 25
  4. 2025VFlowOpt: A Token Pruning Framework for LMMs with Visual Information Flow-Guided Optimizationcited by 22
  5. 2025Efficient Multi-modal Large Language Models via Progressive Consistency Distillationcited by 22
  6. 2025InternVL-X: Advancing and Accelerating InternVL Series with Efficient Visual Token Compressioncited by 21
  7. 2025Balanced Token Pruning: Accelerating Vision Language Models Beyond Local Optimizationcited by 20
  8. 2025FastDriveVLA: Efficient End-to-End Driving via Plug-and-Play Reconstruction-based Token Pruningcited by 20
  9. 2025TokenCarve: Information-Preserving Visual Token Compression in Multimodal Large Language Modelscited by 19
  10. 2025MMS-LLaMA: Efficient LLM-based Audio-Visual Speech Recognition with Minimal Multimodal Speech Tokenscited by 19
  11. 2025Task-Oriented Feature Compression for Multimodal Understanding via Device-Edge Co-Inferencecited by 18
  12. 2025FlexSelect: Flexible Token Selection for Efficient Long Video Understandingcited by 18
  13. 2025Stream-Omni: Simultaneous Multimodal Interactions with Large Language-Vision-Speech Modelcited by 18
  14. 2026FlashVID: Efficient Video Large Language Models via Training-free Tree-based Spatiotemporal Token Mergingcited by 18
  15. 2026HiDrop: Hierarchical Vision Token Reduction in MLLMs via Late Injection, Concave Pyramid Pruning, and Early Exitcited by 18
  16. 2025VisiPruner: Decoding Discontinuous Cross-Modal Dynamics for Efficient Multimodal LLMscited by 17
  17. 2025Less Is More, but Where? Dynamic Token Compression via LLM-Guided Keyframe Priorcited by 15
  18. 2025CROP: Contextual Region-Oriented Visual Token Pruningcited by 14
  19. 2025Visual Multi-Agent System: Mitigating Hallucination Snowballing via Visual Flowcited by 13
  20. 2025VideoScan: Enabling Efficient Streaming Video Understanding via Frame-level Semantic Carrierscited by 12
  21. 2025Retrv-R1: A Reasoning-Driven MLLM Framework for Universal and Efficient Multimodal Retrievalcited by 12
  22. 2025Adaptive Audio-Visual Speech Recognition via Matryoshka-Based Multimodal LLMscited by 11
  23. 2025PUMA: Layer-Pruned Language Model for Efficient Unified Multimodal Retrieval with Modality-Adaptive Learningcited by 11
  24. 2025HiPrune: Hierarchical Attention for Efficient Token Pruning in Vision-Language Modelscited by 11
  25. 2026CoPE-VideoLM: Leveraging Codec Primitives For Efficient Video Language Modelingcited by 11
  26. 2025DyMU: Dynamic Merging and Virtual Unmerging for Efficient VLMscited by 10
  27. 2025Map the Flow: Revealing Hidden Pathways of Information in VideoLLMscited by 10
  28. 2025MVU-Eval: Towards Multi-Video Understanding Evaluation for Multimodal LLMscited by 10
  29. 2026Embodied Foundation Models at the Edge: A Survey of Deployment Constraints and Mitigation Strategiescited by 9
  30. 2026Do 3D Large Language Models Really Understand 3D Spatial Relationships?cited by 9
  31. 2025SAMA: Towards Multi-Turn Referential Grounded Video Chat with Large Language Modelscited by 8
  32. 2025IV-Bench: A Benchmark for Image-Grounded Video Perception and Reasoning in Multimodal LLMscited by 7
  33. 2025One Trajectory, One Token: Grounded Video Tokenization via Panoptic Sub-object Trajectorycited by 7
  34. 2025DynTok: Dynamic Compression of Visual Tokens for Efficient and Effective Video Understandingcited by 7
  35. 2025ElasticMM: Efficient Multimodal LLMs Serving with Elastic Multimodal Parallelismcited by 7
  36. 2025MergeMix: A Unified Augmentation Paradigm for Visual and Multi-Modal Understandingcited by 7
  37. 2025CARES: Context-Aware Resolution Selector for VLMscited by 6
  38. 2025FLoC: Facility Location-Based Efficient Visual Token Compression for Long Video Understandingcited by 6

50 most cited of 166 All 166 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024iLLaVA: An Image is Worth Fewer Than 1/3 Input Tokens in Large Multimodal Models
  2. 2025Rethinking Visual Information Processing in Multimodal LLMs
  3. 2025LLaVA-OneVision-1.5: Fully Open Framework for Democratized Multimodal Training
  4. 2024TS-LLaVA: Constructing Visual Tokens through Thumbnail-and-Sampling for Training-Free Video Large Language Models
  5. 2024LLaVA-OneVision: Easy Visual Task Transfer
  6. 2025Can Visual Input Be Compressed? A Visual Token Compression Benchmark for Large Multimodal Models
  7. 2024LLaVA-UHD: an LMM Perceiving Any Aspect Ratio and High-Resolution Images
  8. 2024AVG-LLaVA: An Efficient Large Multimodal Model with Adaptive Visual Granularity