# MEDA: Dynamic KV Cache Allocation for Efficient Multimodal Long-Context Inference

arXiv: https://arxiv.org/abs/2502.17599 · HTML: https://mlsanity.com/paper/2502.17599/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- Placed in Visual token pruning and efficiency, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 35 of its 61 references that are in the map.
- First posted on arXiv 24 Feb 2025.

## Placed in

- Multimodal LLMs / vision-language models / [Visual token pruning and efficiency](https://mlsanity.com/mllm/visual-token-pruning-and-efficiency/)
- Efficient LLM inference and serving / [KV cache optimization](https://mlsanity.com/inference/kv-cache-optimization/)

## Builds on (most cited first, 30 of 35)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness](https://mlsanity.com/paper/2205.14135/) (2022)
- [InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks](https://mlsanity.com/paper/2312.14238/) (2023)
- [Efficient Streaming Language Models with Attention Sinks](https://mlsanity.com/paper/2309.17453/) (2023)
- [SnapKV: LLM Knows What You are Looking for Before Generation](https://mlsanity.com/paper/2404.14469/) (2024)
- [H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models](https://mlsanity.com/paper/2306.14048/) (2023)
- [KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache](https://mlsanity.com/paper/2402.02750/) (2024)
- [LLaVA-Video: Video Instruction Tuning With Synthetic Data](https://mlsanity.com/paper/2410.02713/) (2024)
- [Long Context Transfer from Language to Vision](https://mlsanity.com/paper/2406.16852/) (2024)
- [PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funneling](https://mlsanity.com/paper/2406.02069/) (2024)
- [LongVILA: Scaling Long-Context Visual Language Models for Long Videos](https://mlsanity.com/paper/2408.10188/) (2024)
- [SVD-LLM: Truncation-aware Singular Value Decomposition for Large Language Model Compression](https://mlsanity.com/paper/2403.07378/) (2024)
- [Video-XL: Extra-Long Vision Language Model for Hour-Scale Video Understanding](https://mlsanity.com/paper/2409.14485/) (2024)
- [World Model on Million-Length Video And Language With Blockwise RingAttention](https://mlsanity.com/paper/2402.08268/) (2024)
- [Stabilizing Transformer Training by Preventing Attention Entropy Collapse](https://mlsanity.com/paper/2303.06296/) (2023)
- [GEAR: An Efficient KV Cache Compression Recipe for Near-Lossless Generative Inference of LLM](https://mlsanity.com/paper/2403.05527/) (2024)
- [PyramidInfer: Pyramid KV Cache Compression for High-throughput LLM Inference](https://mlsanity.com/paper/2405.12532/) (2024)
- [Tarsier: Recipes for Training and Evaluating Large Video Description Models](https://mlsanity.com/paper/2407.00634/) (2024)
- [Scaling Laws with Vocabulary: Larger Models Deserve Larger Vocabularies](https://mlsanity.com/paper/2407.13623/) (2024)
- [LongLLaVA: Scaling Multi-modal LLMs to 1000 Images Efficiently via a Hybrid Architecture](https://mlsanity.com/paper/2409.02889/) (2024)
- [LOOK-M: Look-Once Optimization in KV Cache for Efficient Multimodal Long-Context Inference](https://mlsanity.com/paper/2406.18139/) (2024)
- [LLaMA-Berry: Pairwise Optimization for O1-like Olympiad-Level Mathematical Reasoning](https://mlsanity.com/paper/2410.02884/) (2024)
- [MileBench: Benchmarking MLLMs in Long Context](https://mlsanity.com/paper/2404.18532/) (2024)
- [MADTP: Multimodal Alignment-Guided Dynamic Token Pruning for Accelerating Vision-Language Transformer](https://mlsanity.com/paper/2403.02991/) (2024)
- [Autoregressive Models in Vision: A Survey](https://mlsanity.com/paper/2411.05902/) (2024)
- [Recent Advances in Large Langauge Model Benchmarks against Data Contamination: From Static to Dynamic Evaluation](https://mlsanity.com/paper/2502.17521/) (2025)
- [Efficient Diffusion Models: A Survey](https://mlsanity.com/paper/2502.06805/) (2025)
- [On the Efficacy of Eviction Policy for Key-Value Constrained Generative Language Model Inference](https://mlsanity.com/paper/2402.06262/) (2024)
- [MEIT: Multimodal Electrocardiogram Instruction Tuning on Large Language Models for Report Generation](https://mlsanity.com/paper/2403.04945/) (2024)
- [NeuroClips: Towards High-fidelity and Smooth fMRI-to-Video Reconstruction](https://mlsanity.com/paper/2410.19452/) (2024)

## Built on by (most cited first, 30 of 31)

- [SVD-LLM V2: Optimizing Singular Value Truncation for Large Language Model Compression](https://mlsanity.com/paper/2503.12340/) (2025)
- [SRPO: Enhancing Multimodal LLM Reasoning via Reflection-Aware Reinforcement Learning](https://mlsanity.com/paper/2506.01713/) (2025)
- [Look-Back: Implicit Visual Re-focusing in MLLM Reasoning](https://mlsanity.com/paper/2507.03019/) (2025)
- [Shifting AI Efficiency From Model-Centric to Data-Centric Compression](https://mlsanity.com/paper/2505.19147/) (2025)
- [Head-Aware KV Cache Compression for Efficient Visual Autoregressive Modeling](https://mlsanity.com/paper/2504.09261/) (2025)
- [GUI-KV: Efficient GUI Agents via KV Cache with Spatio-Temporal Awareness](https://mlsanity.com/paper/2510.00536/) (2025)
- [Make Each Token Count: Towards Improving Long-Context Performance with KV Cache Eviction](https://mlsanity.com/paper/2605.09649/) (2026)
- [Efficient Inference for Large Vision-Language Models: Bottlenecks, Techniques, and Prospects](https://mlsanity.com/paper/2604.05546/) (2026)
- [LightVLM: Acceleraing Large Multimodal Models with Pyramid Token Merging and KV Cache Compression](https://mlsanity.com/paper/2509.00419/) (2025)
- [Event-VStream: Event-Driven Real-Time Understanding for Long Video Streams](https://mlsanity.com/paper/2601.15655/) (2026)
- [Enhancing Test-Time Scaling of Large Language Models with Hierarchical Retrieval-Augmented MCTS](https://mlsanity.com/paper/2507.05557/) (2025)
- [Revisiting Multimodal KV Cache Compression: A Frequency-Domain-Guided Outlier-KV-Aware Approach](https://mlsanity.com/paper/2511.16786/) (2025)
- [HybridKV: Hybrid KV Cache Compression for Efficient Multimodal Large Language Model Inference](https://mlsanity.com/paper/2604.05887/) (2026)
- [Prefill-Time Intervention for Mitigating Hallucination in Large Vision-Language Models](https://mlsanity.com/paper/2604.25642/) (2026)
- [RetentiveKV: State-Space Memory for Uncertainty-Aware Multimodal KV Cache Eviction](https://mlsanity.com/paper/2605.04075/) (2026)
- [PRISM: Pareto-Efficient Retrieval over Intent-Aware Structured Memory for Long-Horizon Agents](https://mlsanity.com/paper/2605.12260/) (2026)
- [Adaptive and Resource-efficient Agentic AI Systems for Mobile and Embedded Devices: A Survey](https://mlsanity.com/paper/2510.00078/) (2025)
- [ST-Lite: Training-Free KV Cache Compression with Spatio-Trajectory Guidance for Long-Horizon GUI Agents](https://mlsanity.com/paper/2603.00188/) (2026)
- [ERA: Entropy-Guided Visual Token Pruning with Rectified Attention for Efficient MLLMs](https://mlsanity.com/paper/2606.31982/) (2026)
- [Dynamic Token Reweighting for Robust Vision-Language Models](https://mlsanity.com/paper/2505.17132/) (2025)
- [SpecEyes: Accelerating Agentic Multimodal LLMs via Speculative Perception and Planning](https://mlsanity.com/paper/2603.23483/) (2026)
- [Linear Scaling Video VLMs for Long Video Understanding](https://mlsanity.com/paper/2605.31598/) (2026)
- [STaR-KV: Spatio-Temporal Adaptive Re-weighting for KV Cache Compression in GUI Vision-Language Models](https://mlsanity.com/paper/2606.01790/) (2026)
- [HACK++: Towards More Effective Head-Aware Key-Value Compression for Efficient Visual Autoregressive Modeling](https://mlsanity.com/paper/2606.08302/) (2026)
- [Reducing Peak Memory Usage for Modern Multimodal Large Language Model Pipelines](https://mlsanity.com/paper/2604.16734/) (2026)
- [Learning to See What You Need: Gaze Attention for Multimodal Large Language Models](https://mlsanity.com/paper/2605.13080/) (2026)
- [VisCache: Visual KV Cache Pruning for Efficient Vision Large Language Model Inference](https://mlsanity.com/paper/2608.24063/) (2026)
- [CONDUIT: A Unified Residual-Stream Restoration Framework for KV Cache Reuse in Vision-Language Models](https://mlsanity.com/paper/2609.05821/) (2026)
- [ECOKV: Geometry-Aware KV Cache Eviction via Complementary Diversity Metrics](https://mlsanity.com/paper/2609.06663/) (2026)
- [Why Is Video Still So Expensive? A Survey of Inference-Efficiency Mechanisms in Video and Audiovisual LLMs](https://mlsanity.com/paper/2609.10355/) (2026)

## Abstract (arXiv)

Long-context Multimodal Large Language Models (MLLMs) that incorporate long text-image and text-video modalities, demand substantial resources as their multimodal Key-Value (KV) caches grow with increasing input lengths, challenging inference efficiency. Existing methods for KV cache compression, in both text-only and multimodal LLMs, have neglected attention density variations across layers, thus often adopting uniform or progressive reduction strategies for layer-wise cache allocation. In this work, we propose MEDA, a dynamic layer-wise KV cache allocation method for efficient multimodal long-context inference. As its core, MEDA utilizes cross-modal attention entropy to determine the KV cache size at each MLLMs layer. Given the dynamically allocated KV cache size at each layer, MEDA also employs a KV pair selection scheme to identify which KV pairs to select and a KV pair merging strategy that merges the selected and non-selected ones to preserve information from the entire context. MEDA achieves up to 72% KV cache memory reduction and 2.82 times faster decoding speed, while maintaining or enhancing performance on various multimodal tasks in long-context settings, including multi-images and long-video scenarios. Our code is released at https://github.com/AIoT-MLSys-Lab/MEDA.