Residual-Mass Accounting for Partial-KV Decoding
Not on the map. Why?
Abstract (arXiv)
We study a controlled partial-KV decoding setting in which exact unnormalized softmax contributions are computed for sink/tail anchors and a retrieved token set, while the remaining prefill tokens are represented by a residual estimate. We focus on the accounting rule after the query-dependent exact support has been selected, and use exhaustive Top-K only as an oracle selector, not as a deployable retrieval system. The proposed rule leaves the backbone language model and the exact-branch KV tensors unchanged. It builds fixed-size summary states (S,u) from learned positive feature maps φ, subtracts retrieved-token feature contributions to keep the exact and residual sets non-overlapping, and merges the estimated residual numerator and denominator with the exact branch under one normalization. At a 1% exact-support budget, our residual-completion method improves over the selection-only Top-K baseline on RULER and BABILong across frozen 1B and 3B Llama-3.2-Instruct backbones at all reported context lengths. In the 0.5-4% exact-support budget sweeps, this trend largely persists. On LongBench, summarization results are mostly favorable, while multi-document QA is mixed. Attention-output diagnostics support retrieved-token subtraction as the partition-consistent accounting rule, while indicating that the main remaining error is imperfect learned-φ approximation of the unretrieved residual mass.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 32 of its 36 references that are in the map. First posted on arXiv 7 Apr 2026.
Builds on
32 of 36 references.
References · 32
- 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
- 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
- 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 2,022
- 2020Rethinking Attention with Performerscited by 1,601
- 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,417
- 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,346
- 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,160
- 2019What Does BERT Look At? An Analysis of BERT's Attentioncited by 1,003
- 2024SnapKV: LLM Knows What You are Looking for Before Generationcited by 880
- 2019Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Prunedcited by 856
- 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Modelscited by 832
- 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 575
20 more
- 2024Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inferencecited by 501
- 2018A Discourse-Aware Attention Model for Abstractive Summarization of Long Documentscited by 478
- 2023Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Timecited by 439
- 2024PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funnelingcited by 432
- 2023Prompt Cache: Modular Attention Reuse for Low-Latency Inferencecited by 268
- 2024InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Managementcited by 245
- 2024BABILong: Testing the Limits of LLMs with Long Context Reasoning-in-a-Haystackcited by 215
- 2024Leave No Context Behind: Efficient Infinite Context Transformers with Infini-attentioncited by 200
- 2019BIGPATENT: A Large-Scale Dataset for Abstractive and Coherent Summarizationcited by 156
- 2025LMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inferencecited by 126
- 2024The Hedgehog & the Porcupine: Expressive Linear Attentions with Softmax Mimicrycited by 120
- 2024RetrievalAttention: Accelerating Long-Context LLM Inference via Vector Retrievalcited by 116
- 2024MagicPIG: LSH Sampling for Efficient LLM Generationcited by 111
- 2024AI and Memory Wallcited by 92
- 2021Memory-efficient Transformers via Top-k Attentioncited by 60
- 2025RADLADS: Rapid Attention Distillation to Linear Attention Decoders at Scalecited by 24
- 2022Improved Random Features for Dot Product Kernelscited by 12
- 2025LoLA: Low-Rank Linear Attention With Sparse Cachingcited by 12
- 2025LAWCAT: Efficient Distillation from Quadratic to Linear Attention with Convolution across Tokens for Long Context Modelingcited by 3
- 2026SPLA: Block Sparse Plus Linear Attention for Long Context Modelingcited by 3
Built on by
Citing papers per year: 2026: 1.
1 paper.