mlsanityData to 14 Oct 2026

KV Cache Offloading for Context-Intensive Tasks

arXiv 2604.08426 ↗First posted 9 Apr 2026cs.LG cs.AI cs.CLcited by 1

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

With the growing demand for long-context LLMs across a wide range of applications, the key-value (KV) cache has become a critical bottleneck for both latency and memory usage. Recently, KV-cache offloading has emerged as a promising approach to reduce memory footprint and inference latency while preserving accuracy. Prior evaluations have largely focused on tasks that do not require extracting large amounts of information from the context. In this work, we study KV-cache offloading on context-intensive tasks: problems where the solution requires looking up a lot of information from the input prompt. We create and release the Text2JSON benchmark, a highly context-intensive task that requires extracting structured knowledge from raw text. We evaluate modern KV offloading on Text2JSON and other context-intensive tasks and find significant performance degradation on both Llama 3 and Qwen 3 models. Our analysis identifies two key reasons for poor accuracy: low-rank projection of keys and unreliable landmarks, and proposes a simpler alternative strategy that significantly improves accuracy across multiple LLM families and benchmarks. These findings highlight the need for a comprehensive and rigorous evaluation of long-context compression techniques.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 75 of its 103 references that are in the map. First posted on arXiv 9 Apr 2026.

Builds on

75 of 103 references.

Closest ancestors · 22

Same direction, newest first

  1. 2026Understanding the Physics of Key-Value Cache Compression for LLMs through Attention Dynamics2 Mar
  2. 2026KVzap: Fast, Adaptive, and Faithful KV Cache Pruning12 Jan
  3. 2025Reconstructing KV Caches with Cross-layer Fusion For Enhanced Transformers3 Dec
  4. 2025SnapStream: Efficient Long Sequence Decoding on Dataflow Accelerators5 Nov
  5. 2025The Pitfalls of KV Cache Compression30 Sep
  6. 2025LAVa: Layer-wise KV Cache Eviction with Dynamic Budget Allocation11 Sep
  7. 2025HCAttention: Extreme KV Cache Compression via Heterogeneous Attention Computing for LLMs26 Jul
  8. 2025CAKE: Cascading and Adaptive KV Cache Eviction with Layer Preferences16 Mar
  9. 2025Progressive Sparse Attention: Algorithm and System Co-design for Efficient Attention in LLM Serving1 Mar
  10. 2025Cache Me If You Must: Adaptive Key-Value Quantization for Large Language Models31 Jan
  11. 2024SCBench: A KV Cache-Centric Analysis of Long-Context Methods13 Dec
  12. 2024ShadowKV: KV Cache in Shadows for High-Throughput Long-Context LLM Inference28 Oct
10 more
  1. 2024KVSharer: Efficient Inference via Layer-Wise Dissimilar KV Cache Sharing24 Oct
  2. 2024A Systematic Study of Cross-Layer KV Sharing for Efficient LLM Inference18 Oct
  3. 2024SwiftKV: Fast Prefill-Optimized Inference with Knowledge-Preserving Model Transformation4 Oct
  4. 2024Ada-KV: Optimizing KV Cache Eviction by Adaptive Budget Allocation for Efficient LLM Inference16 Jul
  5. 2024InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Management28 Jun
  6. 2024Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference16 Jun
  7. 2024PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funneling4 Jun
  8. 2024SnapKV: LLM Knows What You are Looking for Before Generation22 Apr
  9. 2023Efficient Memory Management for Large Language Model Serving with PagedAttention12 Sep
  10. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models24 Jun

Other references · 53

  1. 2017Attention Is All You Needcited by 60,361
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  5. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,718
  6. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  7. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,417
  8. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,346
  9. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,346
  10. 2024Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Dualitycited by 1,336
  11. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,141
  12. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,135
38 more
  1. 2024ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Toolscited by 1,121
  2. 2024LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memorycited by 707
  3. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 637
  4. 2024QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMscited by 634
  5. 2023YaRN: Efficient Context Window Extension of Large Language Modelscited by 601
  6. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantizationcited by 591
  7. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 575
  8. 2021FinQA: A Dataset of Numerical Reasoning over Financial Datacited by 562
  9. 2024Gated Delta Networks: Improving Mamba2 with Delta Rulecited by 532
  10. 2022DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scalecited by 407
  11. 2025Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attentioncited by 407
  12. 2024Jamba: A Hybrid Transformer-Mamba Language Modelcited by 401
  13. 2024MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attentioncited by 383
  14. 2024LongBench v2: Towards Deeper Understanding and Reasoning on Realistic Long-context Multitaskscited by 379
  15. 2024∞Bench: Extending Long Context Evaluation Beyond 100K Tokenscited by 367
  16. 2023RepoBench: Benchmarking Repository-Level Code Auto-Completion Systemscited by 353
  17. 2023L-Eval: Instituting Standardized Evaluation for Long Context Language Modelscited by 233
  18. 2024BABILong: Testing the Limits of LLMs with Long Context Reasoning-in-a-Haystackcited by 215
  19. 2023Large Language Models for Generative Information Extraction: A Surveycited by 185
  20. 2022SCROLLS: Standardized CompaRison Over Long Language Sequencescited by 170
  21. 2024Leave No Document Behind: Benchmarking Long-Context LLMs with Extended Multi-Doc QAcited by 122
  22. 2025SpargeAttention: Accurate and Training-free Sparse Attention Accelerating Any Model Inferencecited by 120
  23. 2024RetrievalAttention: Accelerating Long-Context LLM Inference via Vector Retrievalcited by 116
  24. 2024MagicPIG: LSH Sampling for Efficient LLM Generationcited by 111
  25. 2024PQCache: Product Quantization-based KVCache for Long Context LLM Inferencecited by 95
  26. 2024Loki: Low-rank Keys for Efficient Sparse Attentioncited by 89
  27. 2024LV-Eval: A Balanced Long-Context Benchmark with 5 Length Levels Up to 256Kcited by 81
  28. 2024RepoAgent: An LLM-Powered Open-Source Framework for Repository-level Code Documentation Generationcited by 70
  29. 2021GenIE: Generative Information Extractioncited by 53
  30. 2023BAMBOO: A Comprehensive Benchmark for Evaluating Long Text Modeling Capacities of Large Language Modelscited by 53
  31. 2018Neural Open Information Extractioncited by 50
  32. 2025Twilight: Adaptive Attention Sparsity with Hierarchical Top-p Pruningcited by 49
  33. 2024NeedleBench: Evaluating LLM Retrieval and Reasoning Across Varying Information Densitiescited by 32
  34. 2024Fast Matrix Multiplications for Lookup Table-Quantized LLMscited by 29
  35. 2025LongProc: Benchmarking Long-Context Language Models on Long Procedural Generationcited by 29
  36. 2024Pushing the Limits of Large Language Model Quantization via the Linearity Theoremcited by 21
  37. 2021Transformer Acceleration with Dynamic Sparse Attentioncited by 19
  38. 2023Marathon: A Race Through the Realm of Long Context with Large Language Modelscited by 9

50 most cited of 53 All 53 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026When Does Value-Aware KV Eviction Help? A Fixed-Contract Diagnostic for Non-Monotone Cache Compressioncited by 5

Similar, not cited

Close in wording, no citation either way

  1. 2025KV Cache Compression for Inference Efficiency in LLMs: A Review
  2. 2025Cache Me If You Can: How Many KVs Do You Need for Effective Long-Context LMs?
  3. 2024KV Cache Compression, But What Must We Give in Return? A Comprehensive Benchmark of Long Context Capable Approaches
  4. 2024KV-Compress: Paged KV-Cache Compression with Variable Compression Rates per Attention Head
  5. 2025Key, Value, Compress: A Systematic Exploration of KV Cache Compression Techniques
  6. 2026KV Cache Optimization Strategies for Scalable and Efficient LLM Inference
  7. 2026Comparative Characterization of KV Cache Management Strategies for LLM Inference
  8. 2025BaKlaVa -- Budgeted Allocation of KV cache for Long-context Inference