mlsanityData to 7 Oct 2026

Hierarchical Adaptive Eviction for KV Cache Management in Multimodal Language Models

arXiv 2602.02197 ↗First posted 2 Feb 2026cs.LG cs.AIcited by 0

On the map: Multimodal LLMs / vision-language models → Visual token pruning and efficiency

also in 1 more
Abstract (arXiv)

The integration of visual information into Large Language Models (LLMs) has enabled Multimodal LLMs (MLLMs), but the quadratic memory and computational costs of Transformer architectures remain a bottleneck. Existing KV cache eviction strategies fail to address the heterogeneous attention distributions between visual and text tokens, leading to suboptimal efficiency or degraded performance. In this paper, we propose Hierarchical Adaptive Eviction (HAE), a KV cache eviction framework that optimizes text-visual token interaction in MLLMs by implementing Dual-Attention Pruning during pre-filling (leveraging visual token sparsity and attention variance) and a Dynamic Decoding Eviction Strategy (inspired by OS Recycle Bins) during decoding. HAE minimizes KV cache usage across layers, reduces computational overhead via index broadcasting, and theoretically ensures superior information integrity and lower error bounds compared to greedy strategies, enhancing efficiency in both comprehension and generation tasks. Empirically, HAE reduces KV-Cache memory by 41% with minimal accuracy loss (0.3% drop) in image understanding tasks and accelerates story generation inference by 1.5x while maintaining output quality on Phi3.5-Vision-Instruct model.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Visual token pruning and efficiency, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 25 of its 29 references that are in the map. First posted on arXiv 2 Feb 2026.

Builds on

25 of 29 references.

Closest ancestors · 8

Same direction, newest first

  1. 2024Dynamic-LLaVA: Efficient Multimodal Large Language Models via Dynamic Vision-language Context Sparsification1 Dec
  2. 2024Multi-Stage Vision Token Dropping: Towards Efficient Multimodal Large Language Model16 Nov
  3. 2024SparseVLM: Visual Token Sparsification for Efficient Vision-Language Model Inference6 Oct
  4. 2024HiRED: Attention-Guided Token Dropping for Efficient Inference of High-Resolution Vision-Language Models20 Aug
  5. 2024TokenPacker: Efficient Visual Projector for Multimodal LLM2 Jul
  6. 2024VoCo-LLaMA: Towards Vision Compression with Large Language Models18 Jun
  7. 2024DeCo: Decoupling Token Compression from Semantic Abstraction in Multimodal Large Language Models31 May
  8. 2024An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Models11 Mar

Other references · 17

  1. 2017Attention Is All You Needcited by 59,932
  2. 2023Visual Instruction Tuningcited by 7,792
  3. 2023Qwen Technical Reportcited by 3,273
  4. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  5. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,883
  6. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,461
  7. 2022Token Merging: Your ViT But Fastercited by 995
  8. 2023Accelerating Large Language Model Decoding with Speculative Samplingcited by 975
  9. 2024SnapKV: LLM Knows What You are Looking for Before Generationcited by 848
  10. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Modelscited by 805
  11. 2022Efficiently Scaling Transformer Inferencecited by 529
  12. 2024FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precisioncited by 507
5 more
  1. 2017TGIF-QA: Toward Spatio-Temporal Reasoning in Visual Question Answeringcited by 378
  2. 2024Ada-KV: Optimizing KV Cache Eviction by Adaptive Budget Allocation for Efficient LLM Inferencecited by 230
  3. 2024A Survey on Efficient Inference for Large Language Modelscited by 197
  4. 2024SEED-Story: Multimodal Long Story Generation with Large Language Modelcited by 70
  5. 2024NACL: A General and Effective KV Cache Eviction Framework for LLMs at Inference Timecited by 43

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026RetentiveKV: State-Space Memory for Uncertainty-Aware Multimodal KV Cache Eviction
  2. 2025CAKE: Cascading and Adaptive KV Cache Eviction with Layer Preferences
  3. 2026HybridKV: Hybrid KV Cache Compression for Efficient Multimodal Large Language Model Inference
  4. 2026ReST-KV: Robust KV Cache Eviction with Layer-wise Output Reconstruction and Spatial-Temporal Smoothing
  5. 2025MEDA: Dynamic KV Cache Allocation for Efficient Multimodal Long-Context Inference
  6. 2024LOOK-M: Look-Once Optimization in KV Cache for Efficient Multimodal Long-Context Inference
  7. 2026Reducing Peak Memory Usage for Modern Multimodal Large Language Model Pipelines
  8. 2026Reformulating KV Cache Eviction Problem for Long-Context LLM Inference