mlsanityData to 14 Oct 2026

LOOK-M: Look-Once Optimization in KV Cache for Efficient Multimodal Long-Context Inference

arXiv 2406.18139 ↗First posted 26 Jun 2024cs.CL cs.CVcited by 95

On the map: Multimodal LLMs / vision-language models → Visual token pruning and efficiency

also in 1 more
Abstract (arXiv)

Long-context Multimodal Large Language Models (MLLMs) demand substantial computational resources for inference as the growth of their multimodal Key-Value (KV) cache, in response to increasing input lengths, challenges memory and time efficiency. Unlike single-modality LLMs that manage only textual contexts, the KV cache of long-context MLLMs includes representations from multiple images with temporal and spatial relationships and related textual contexts. The predominance of image tokens means traditional optimizations for LLMs' KV caches are unsuitable for multimodal long-context settings, and no prior works have addressed this challenge. In this work, we introduce LOOK-M, a pioneering, fine-tuning-free approach that efficiently reduces the multimodal KV cache size while maintaining performance comparable to a full cache. We observe that during prompt prefill, the model prioritizes more textual attention over image features, and based on the multimodal interaction observation, a new proposed text-prior method is explored to compress the KV cache. Furthermore, to mitigate the degradation of image contextual information, we propose several compensatory strategies using KV pairs merging. LOOK-M demonstrates that with a significant reduction in KV Cache memory usage, such as reducing it by 80% in some cases, it not only achieves up to 1.5x faster decoding but also maintains or even enhances performance across a variety of long context multimodal tasks.

Key facts. Cited by 95 papers in the map (data through 14 Oct 2026). Placed in Visual token pruning and efficiency, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 31 of its 61 references that are in the map. First posted on arXiv 26 Jun 2024.

Builds on

31 of 61 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024LLaVA-PruMerge: Adaptive Token Reduction for Efficient Large Multimodal Models22 Mar
  2. 2024MoPE-CLIP: Structured Pruning for Efficient Vision-Language Models with Module-wise Pruning Error Metric12 Mar

Other references · 29

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2023GPT-4 Technical Reportcited by 18,396
  4. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  5. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,565
  6. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  7. 2022Token Merging: Your ViT But Fastercited by 1,021
  8. 2024SnapKV: LLM Knows What You are Looking for Before Generationcited by 880
  9. 2023The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)cited by 681
  10. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 637
  11. 2023Model Tells You What to Discard: Adaptive KV Cache Compression for LLMscited by 474
  12. 2021SPViT: Enabling Faster Vision Transformers via Soft Token Pruningcited by 167
17 more
  1. 2024Dynamic Memory Compression: Retrofitting LLMs for Accelerated Inferencecited by 116
  2. 2024No Token Left Behind: Reliable KV Cache Compression via Importance-Aware Mixed Precision Quantizationcited by 101
  3. 2024MileBench: Benchmarking MLLMs in Long Contextcited by 88
  4. 2023Joint Token Pruning and Squeezing Towards More Aggressive Compression of Vision Transformerscited by 81
  5. 2024Get More with LESS: Synthesizing Recurrence with KV Cache Compression for Efficient LLM Inferencecited by 75
  6. 2024Zero-Shot ECG Classification with Multimodal Learning and Test-time Clinical Knowledge Enhancementcited by 67
  7. 2023Dynamic Token Pruning in Plain Vision Transformers for Semantic Segmentationcited by 62
  8. 2023Med-UniC: Unifying Cross-Lingual Medical Vision-Language Pre-Training by Diminishing Biascited by 57
  9. 2024MADTP: Multimodal Alignment-Guided Dynamic Token Pruning for Accelerating Vision-Language Transformercited by 54
  10. 2024IntactKV: Improving Large Language Model Quantization by Keeping Pivot Tokens Intactcited by 52
  11. 2022CP-ViT: Cascade Vision Transformer Pruning via Progressive Sparsity Predictioncited by 39
  12. 2024On the Efficacy of Eviction Policy for Key-Value Constrained Generative Language Model Inferencecited by 32
  13. 2023ETP: Learning Transferable ECG Representations via ECG-Text Pre-trainingcited by 29
  14. 2022G-MAP: General Memory-Augmented Pre-trained Language Model for Domain Taskscited by 9
  15. 2024Focus on the Core: Efficient Attention via Pruned Token Compression for Document Classificationcited by 7
  16. 2020Large Scale Many-Objective Optimization Driven by Distributional Adversarial Networkscited by 6
  17. 2020Many-Objective Estimation of Distribution Optimization Algorithm Based on WGAN-GPcited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 17; 2025: 39; 2026: 39.

95 papers.

Sort:

By area:

  1. 2024MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attentioncited by 383
  2. 2024When Attention Sink Emerges in Language Models: An Empirical Viewcited by 208
  3. 2024A Survey on Large Language Model Acceleration based on KV Cache Managementcited by 131
  4. 2024DyCoke: Dynamic Compression of Tokens for Fast Video Large Language Modelscited by 96
  5. 2025SRPO: Enhancing Multimodal LLM Reasoning via Reflection-Aware Reinforcement Learningcited by 57
  6. 2025AdaReTaKe: Adaptive Redundancy Reduction to Perceive Longer for Video-language Understandingcited by 48
  7. 2024ZipVL: Efficient Large Vision-Language Models with Dynamic Token Sparsificationcited by 44
  8. 2024MoH: Multi-Head Attention as Mixture-of-Head Attentioncited by 44
  9. 2025StreamingTOM: Streaming Token Compression for Efficient Video Understandingcited by 43
  10. 2024Feather the Throttle: Revisiting Visual Token Pruning for Vision-Language Model Accelerationcited by 41
  11. 2025KVTuner: Sensitivity-Aware Layer-Wise Mixed-Precision KV Cache Quantization for Efficient and Nearly Lossless LLM Inferencecited by 40
  12. 2024Discovering the Gems in Early Layers: Accelerating Long-Context LLMs with 1000x Input Token Reductioncited by 39
38 more
  1. 2024xGen-MM-Vid (BLIP-3-Video): You Only Need 32 Tokens to Represent a Video Even in VLMscited by 34
  2. 2024ReTaKe: Reducing Temporal and Knowledge Redundancy for Long Video Understandingcited by 33
  3. 2025MEDA: Dynamic KV Cache Allocation for Efficient Multimodal Long-Context Inferencecited by 31
  4. 2025Multi-Granular Spatio-Temporal Token Merging for Training-Free Acceleration of Video LLMscited by 31
  5. 2025Shifting AI Efficiency From Model-Centric to Data-Centric Compressioncited by 29
  6. 2025Spatial Understanding from Videos: Structured Prompts Meet Simulation Datacited by 28
  7. 2024Inference Optimal VLMs Need Fewer Visual Tokens and More Parameterscited by 25
  8. 2025Memory-Efficient Visual Autoregressive Modeling with Scale-Aware KV Cache Compressioncited by 22
  9. 2024LinVT: Empower Your Image-level Large Language Model to Understand Videoscited by 19
  10. 2025Head-Aware KV Cache Compression for Efficient Visual Autoregressive Modelingcited by 15
  11. 2025Speculative Prefill: Turbocharging TTFT with Lightweight and Training-Free Token Importance Estimationcited by 14
  12. 2025Mixing Importance with Diversity: Joint Optimization for KV Cache Compression in Large Vision-Language Modelscited by 14
  13. 2024PrefixKV: Adaptive Prefix KV Cache is What Vision Instruction-Following Models Need for Efficient Generationcited by 13
  14. 2025Plug-and-Play 1.x-Bit KV Cache Quantization for Video Large Language Modelscited by 13
  15. 2024Cross-Self KV Cache Pruning for Efficient Vision-Language Inferencecited by 12
  16. 2025MadaKV: Adaptive Modality-Perception KV Cache Eviction for Efficient Multimodal Long-Context Inferencecited by 12
  17. 2025KV Cache Compression for Inference Efficiency in LLMs: A Reviewcited by 12
  18. 2025GUI-KV: Efficient GUI Agents via KV Cache with Spatio-Temporal Awarenesscited by 11
  19. 2025Artificial Hippocampus Networks for Efficient Long-Context Modelingcited by 11
  20. 2025VisionSelector: End-to-End Learnable Visual Token Compression for Efficient Multimodal LLMscited by 11
  21. 2025Efficient LLaMA-3.2-Vision by Trimming Cross-attended Visual Featurescited by 10
  22. 2025EVICPRESS: Joint KV-Cache Compression and Eviction for Efficient LLM Servingcited by 10
  23. 2025AKVQ-VL: Attention-Aware KV Cache Adaptive 2-Bit Quantization for Vision-Language Modelscited by 9
  24. 2026Efficient Inference for Large Vision-Language Models: Bottlenecks, Techniques, and Prospectscited by 9
  25. 2024Famba-V: Fast Vision Mamba with Cross-Layer Token Fusioncited by 8
  26. 2025FlowMM: Cross-Modal Information Flow Guided KV Cache Merging for Efficient Multimodal Context Inferencecited by 6
  27. 2026Event-VStream: Event-Driven Real-Time Understanding for Long Video Streamscited by 6
  28. 2024FastAdaSP: Multitask-Adapted Efficient Inference for Large Speech Language Modelcited by 5
  29. 2025METok: Multi-Stage Event-based Token Compression for Efficient Long Video Understandingcited by 5
  30. 2025Enhancing Test-Time Scaling of Large Language Models with Hierarchical Retrieval-Augmented MCTScited by 5
  31. 2026From Prefix Cache to Fusion RAG Cache: Accelerating LLM Inference in Retrieval-Augmented Generationcited by 5
  32. 2026OScaR: The Occam's Razor for Extreme KV Cache Quantization in LLMs and Beyondcited by 5
  33. 2024Deploying Foundation Model Powered Agent Services: A Surveycited by 4
  34. 2025ACT as Human: Multimodal Large Language Model Data Annotation with Critical Thinkingcited by 4
  35. 2026Geometry-Guided 3D Visual Token Pruning for Video-Language Modelscited by 4
  36. 2026What to Keep, What to Forget: A Rate--Distortion View of Memory Compaction in LLMs and Agentscited by 4
  37. 2025Efficient LLM Inference with Activation Checkpointing and Hybrid Cachingcited by 3
  38. 2025CalibQuant: 1-Bit KV Cache Quantization for Multimodal LLMscited by 3

50 most cited of 95 All 95 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025LLMs Know What to Drop: Self-Attention Guided KV Cache Eviction for Efficient Long-Context Inference
  2. 2026Reformulating KV Cache Eviction Problem for Long-Context LLM Inference
  3. 2025MPIC: Position-Independent Multimodal Context Caching System for Efficient MLLM Serving
  4. 2025BaKlaVa -- Budgeted Allocation of KV cache for Long-context Inference
  5. 2025AirCache: Activating Inter-modal Relevancy KV Cache Compression for Efficient Large Vision-Language Model Inference
  6. 2026Hierarchical Adaptive Eviction for KV Cache Management in Multimodal Language Models
  7. 2024TokenSelect: Efficient Long-Context Inference and Length Extrapolation for LLMs via Dynamic Token-Level KV Cache Selection
  8. 2025KV Pareto: Systems-Level Optimization of KV Cache and Model Compression for Long Context Inference