mlsanityData to 14 Oct 2026

Make Your LVLM KV Cache More Lightweight

arXiv 2605.00789 ↗First posted 1 May 2026cs.CV cs.AI cs.LGcited by 0

On the map: Multimodal LLMs / vision-language models → Visual token pruning and efficiency

also in 1 more
Abstract (arXiv)

Key-Value (KV) cache has become a de facto component of modern Large Vision-Language Models (LVLMs) for inference. While it enhances decoding efficiency in Large Language Models (LLMs), its direct adoption in LVLMs introduces substantial GPU memory overhead due to the large number of vision tokens processed during the prefill stage. To tackle this problem, we propose LightKV, a novel approach that reduces KV cache size by exploiting the redundancy among vision-token embeddings. Guided by text prompts, LightKV employs cross-modality message passing to aggregate informative messages across vision tokens and progressively compress them during prefill. This prompt-aware guidance distinguishes our method from prior vision-only compression strategies. We evaluate LightKV on eight open-source LVLMs across eight public benchmark datasets, e.g., MME and SeedBench. Experimental results demonstrate that with only 55% of the original vision tokens, LightKV (a) halves the vision-token KV cache size, (b) reduces computation by up to 40%, and (c) preserves general-purpose performance while significantly outperforming existing baselines.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Visual token pruning and efficiency, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 74 of its 82 references that are in the map. First posted on arXiv 1 May 2026.

Builds on

74 of 82 references.

Closest ancestors · 6

Same direction, newest first

  1. 2025DivPrune: Diversity-based Visual Token Pruning for Large Multimodal Models4 Mar
  2. 2025LLaVA-Mini: Efficient Image and Video Large Multimodal Models with One Vision Token7 Jan
  3. 2024Efficient Inference of Vision Instruction-Following Models with Elastic Cache25 Jul
  4. 2024Efficient Large Multi-modal Models via Visual Context Compression28 Jun
  5. 2024Matryoshka Query Transformer for Large Vision-Language Models29 May
  6. 2024An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Models11 Mar

Other references · 68

  1. 2017Attention Is All You Needcited by 60,361
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2023GPT-4 Technical Reportcited by 18,396
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  6. 2024The Llama 3 Herd of Modelscited by 14,715
  7. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  8. 2023Visual Instruction Tuningcited by 7,869
  9. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  10. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  11. 2025Qwen2.5-VL Technical Reportcited by 5,587
  12. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
38 more
  1. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  2. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  3. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  4. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  5. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,719
  6. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  7. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,565
  8. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,375
  9. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,224
  10. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,071
  11. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,919
  12. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,729
  13. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,652
  14. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,599
  15. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,471
  16. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,346
  17. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,239
  18. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,050
  19. 2022Token Merging: Your ViT But Fastercited by 1,021
  20. 2024SnapKV: LLM Knows What You are Looking for Before Generationcited by 880
  21. 2021DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsificationcited by 864
  22. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 858
  23. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Modelscited by 832
  24. 2024DeepSeek-VL: Towards Real-World Vision-Language Understandingcited by 733
  25. 2023OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Modelscited by 565
  26. 2023LLaMA-VID: An Image is Worth 2 Tokens in Large Language Modelscited by 546
  27. 2023Language Is Not All You Need: Aligning Perception with Language Modelscited by 543
  28. 2022Efficiently Scaling Transformer Inferencecited by 539
  29. 2022GroupViT: Semantic Segmentation Emerges from Text Supervisioncited by 531
  30. 2018nocaps: novel object captioning at scalecited by 444
  31. 2023Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Timecited by 439
  32. 2024PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funnelingcited by 432
  33. 2021AdaViT: Adaptive Tokens for Efficient Vision Transformercited by 362
  34. 2024LMMs-Eval: Reality Check on the Evaluation of Large Multimodal Modelscited by 353
  35. 2023MultiModal-GPT: A Vision and Language Model for Dialogue with Humanscited by 255
  36. 2024InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Managementcited by 245
  37. 2024Enhancing the Reasoning Ability of Multimodal Large Language Models via Mixed Preference Optimizationcited by 239
  38. 2021Adaptive Token Sampling For Efficient Vision Transformerscited by 183

50 most cited of 68 All 68 (JSON)

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024VL-Cache: Sparsity and Modality-Aware KV Cache Compression for Vision-Language Model Inference Acceleration
  2. 2026KVCapsule: Efficient Sequential KV Cache Compression for Vision-Language Models with Asymmetric Redundancy
  3. 2025LightVLM: Acceleraing Large Multimodal Models with Pyramid Token Merging and KV Cache Compression
  4. 2025AirCache: Activating Inter-modal Relevancy KV Cache Compression for Efficient Large Vision-Language Model Inference
  5. 2025Mixing Importance with Diversity: Joint Optimization for KV Cache Compression in Large Vision-Language Models
  6. 2026VisCache: Visual KV Cache Pruning for Efficient Vision Large Language Model Inference
  7. 2024PrefixKV: Adaptive Prefix KV Cache is What Vision Instruction-Following Models Need for Efficient Generation
  8. 2025KV-Efficient VLA: A Method to Speed up Vision Language Models with RNN-Gated Chunked KV Cache