mlsanityData to 7 Oct 2026

The Residual Stream Is All You Need: On the Redundancy of the KV Cache in Transformer Inference

arXiv 2603.19664 ↗First posted 20 Mar 2026cs.LG cs.AIcited by 4

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

The key-value (KV) cache is widely treated as essential state in transformer inference, and a large body of work engineers policies to compress, evict, or approximate its entries. We prove that this state is entirely redundant: keys and values at every layer are deterministic projections of the residual stream, and recomputing them from a single residual vector per token incurs exactly zero reconstruction error, not approximately, but bit-identically. We verify this across six models from four architecture families (135M to 4B parameters). Cross-task residual patching at every layer produces D_KL = 0 between patched and original output distributions, confirming that the residual stream satisfies a Markov property and is the sole information-carrying state. Removing the cache entirely and recomputing from scratch yields token-identical output under greedy decoding on all models tested. We build on this result with KV-Direct, a bounded-memory inference scheme that checkpoints residual vectors (5 KB per token on Gemma 3-4B) instead of full KV pairs (136 KB), recomputing keys and values on demand. Over 20 conversation turns, KV-Direct holds peak memory at 42 MB while the standard cache grows past 103 MB. Against five eviction baselines (H2O, StreamingLLM, SnapKV, TOVA, window-only), KV-Direct maintains 100% token match at every cache budget; all baselines degrade to 5-28%. A per-operation latency analysis shows recomputation runs up to 5x faster than reading cached tensors at moderate batch sizes. Code is available at https://github.com/Kaleemullahqasim/KV-Direct.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 36 of its 47 references that are in the map. First posted on arXiv 20 Mar 2026.

Builds on

36 of 47 references.

Closest ancestors · 16

Same direction, newest first

  1. 2024KVPR: Efficient LLM Inference with I/O-Aware KV Cache Partial Recomputation26 Nov
  2. 2024Eigen Attention: Attention in Low-Rank Space for KV Cache Compression10 Aug
  3. 2024RazorAttention: Efficient KV Cache Compression Through Retrieval Heads22 Jul
  4. 2024Ada-KV: Optimizing KV Cache Eviction by Adaptive Budget Allocation for Efficient LLM Inference16 Jul
  5. 2024A Simple and Effective L2 Norm-Based Strategy for KV Cache Compression17 Jun
  6. 2024Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference16 Jun
  7. 2024PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funneling4 Jun
  8. 2024MiniCache: KV Cache Compression in Depth Dimension for Large Language Models23 May
  9. 2024Reducing Transformer Key-Value Cache Size with Cross-Layer Attention21 May
  10. 2024PyramidInfer: Pyramid KV Cache Compression for High-throughput LLM Inference21 May
  11. 2024Layer-Condensed KV Cache for Efficient Inference of Large Language Models17 May
  12. 2024SnapKV: LLM Knows What You are Looking for Before Generation22 Apr
4 more
  1. 2024GEAR: An Efficient KV Cache Compression Recipe for Near-Lossless Generative Inference of LLM8 Mar
  2. 2023Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs3 Oct
  3. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models24 Jun
  4. 2023Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Time26 May

Other references · 20

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  3. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  4. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  5. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  6. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  7. 2022In-context Learning and Induction Headscited by 930
  8. 2016Training Deep Nets with Sublinear Memory Costcited by 825
  9. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 700
  10. 2023Towards Automated Circuit Discovery for Mechanistic Interpretabilitycited by 700
  11. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 604
  12. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantizationcited by 571
8 more
  1. 2021Causal Abstractions of Neural Networkscited by 401
  2. 2024You Only Cache Once: Decoder-Decoder Architectures for Language Modelscited by 135
  3. 2024Transformers are Multi-State RNNscited by 120
  4. 2024Loki: Low-rank Keys for Efficient Sparse Attentioncited by 87
  5. 2024What Matters in Transformers? Not All Attention is Neededcited by 83
  6. 2024KV Cache is 1 Bit Per Channel: Efficient Large Language Model Inference with Coupled Quantizationcited by 74
  7. 2024Transformers represent belief state geometry in their residual streamcited by 73
  8. 2024A Survey of Low-bit Large Language Models: Basics, Systems, and Algorithmscited by 30

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 4.

4 papers.

  1. 2026Do Value Vectors in Deep Layers Need Context from the Residual Stream?cited by 1
  2. 2026CoMem: Reusing Transformer Depth across Queries with Persistent Intermediate Residualscited by 1
  3. 2026Storing Is Not Remembering: LSTM-UT and Bounded Gated Memory for Looped Transformerscited by 0
  4. 2026Cache the Encoder Within:Compact, Reusable Memory across LLM Queriescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026How Much Cache Does Reasoning Need? Depth-Cache Tradeoffs in KV-Compressed Transformers
  2. 2026ResidualQuant: KV Cache Quantization for Looped Transformers with 2-Bit Residuals
  3. 2026Codec-Gauge: Learning Compression-Friendly Gauges for Transformer KV Caches
  4. 2024MLKV: Multi-Layer Key-Value Heads for Memory Efficient Transformer Decoding
  5. 2026The Illusion of Equivalence: Systematic FP16 Divergence in KV-Cached Autoregressive Inference
  6. 2026Beyond Token Eviction: Mixed-Dimension Budget Allocation for Efficient KV Cache Compression
  7. 2026LEAP: Layer-wise Exit-Aware Pretraining for Efficient Transformer Inference
  8. 2025LLMCache: Layer-Wise Caching Strategies for Accelerated Reuse in Transformer Inference