LOCKS: Page-Local Compact Key Summaries for Efficient Long-Context Decoding
Not on the map. Why?
Abstract (arXiv)
Serving large language models at long context is bottlenecked by the key-value (KV) cache, which is read at every decode step. We find that attention keys are approximately low-rank within pages. A single low-rank projection shared across pages can miss page-specific directions; fitting a basis to each page better identifies the pages receiving the most attention at comparable stored selector cost. LOCKS stores a rank-r spectral summary per page, reconstructs its within-page logits, and selects pages by log-sum-exp mass without reading candidate keys or values. It stays within about a point of FullKV on LongBench-v1, tracks the read-every-key exact-LSE oracle on RULER down to the smallest budgets, and retains quality furthest under tight budgets on AIME26 and MATH-500. At a 2048-token budget it matches FullKV aggregate quality beyond 100K context while attending about 2% of tokens. Across ranks 2-8, summaries use 4-10% of full-KV bytes. On GH200 with GPU-resident KV, LOCKS reduces complete decode-step time by 1.8× at 512K context. With full KV offloaded to Grace memory, it reaches 3.82-4.22× the faster dense backend's aggregate throughput at 64K-256K by serving larger batches.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 77 of its 88 references that are in the map. First posted on arXiv 27 Jul 2026.
Builds on
77 of 88 references.
References · 77
- 2024The Llama 3 Herd of Modelscited by 14,715
- 2025Qwen3 Technical Reportcited by 8,820
- 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
- 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
- 2023Let's Verify Step by Stepcited by 3,789
- 2020Longformer: The Long-Document Transformercited by 3,413
- 2016Pointer Sentinel Mixture Modelscited by 3,096
- 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
- 2020Big Bird: Transformers for Longer Sequencescited by 1,684
- 2019Generating Long Sequences with Sparse Transformerscited by 1,546
- 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,417
- 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,346
38 more
- 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,346
- 2024ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Toolscited by 1,121
- 2026DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligencecited by 1,040
- 2024SnapKV: LLM Knows What You are Looking for Before Generationcited by 880
- 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Modelscited by 832
- 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 717
- 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 637
- 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantizationcited by 591
- 2024Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inferencecited by 501
- 2025Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attentioncited by 407
- 2024MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attentioncited by 383
- 2024∞Bench: Extending Long Context Evaluation Beyond 100K Tokenscited by 367
- 2024DuoAttention: Efficient Long-Context LLM Inference with Retrieval and Streaming Headscited by 288
- 2024InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Managementcited by 245
- 2024Ada-KV: Optimizing KV Cache Eviction by Adaptive Budget Allocation for Efficient LLM Inferencecited by 235
- 2023Landmark Attention: Random-Access Infinite Context Length for Transformerscited by 213
- 2025MoBA: Mixture of Block Attention for Long-Context LLMscited by 198
- 2024InfLLM: Training-Free Long-Context Extrapolation for LLMs with an Efficient Context Memorycited by 197
- 2024ShadowKV: KV Cache in Shadows for High-Throughput Long-Context LLM Inferencecited by 134
- 2024SeerAttention: Learning Intrinsic Sparse Attention in Your LLMscited by 124
- 2023SparQ Attention: Bandwidth-Efficient LLM Inferencecited by 119
- 2024RetrievalAttention: Accelerating Long-Context LLM Inference via Vector Retrievalcited by 116
- 2024MagicPIG: LSH Sampling for Efficient LLM Generationcited by 111
- 2024No Token Left Behind: Reliable KV Cache Compression via Importance-Aware Mixed Precision Quantizationcited by 101
- 2024PQCache: Product Quantization-based KVCache for Long Context LLM Inferencecited by 95
- 2025KVzip: Query-Agnostic KV Cache Compression with Context Reconstructioncited by 95
- 2024ZipCache: Accurate and Efficient KV Cache Quantization with Salient Token Identificationcited by 93
- 2024Loki: Low-rank Keys for Efficient Sparse Attentioncited by 89
- 2025Expected Attention: KV Cache Compression by Estimating Attention from Future Queries Distributioncited by 74
- 2024Attention Score is not All You Need for Token Importance Indicator in KV Cache Reduction: Value Also Matterscited by 73
- 2024Eigen Attention: Attention in Low-Rank Space for KV Cache Compressioncited by 72
- 2024ClusterKV: Manipulating LLM KV Cache in Semantic Space for Recallable Compressioncited by 65
- 2024Palu: Compressing KV-Cache with Low-Rank Projectioncited by 64
- 2025R-KV: Redundancy-aware KV Cache Compression for Reasoning Modelscited by 54
- 2025Twilight: Adaptive Attention Sparsity with Hierarchical Top-p Pruningcited by 49
- 2025RocketKV: Accelerating Long-Context LLM Inference via Two-Stage KV Cache Compressioncited by 41
- 2024TokenSelect: Efficient Long-Context Inference and Length Extrapolation for LLMs via Dynamic Token-Level KV Cache Selectioncited by 39
- 2026Fast KV Compaction via Attention Matchingcited by 31
50 most cited of 77 All 77 (JSON)
Built on by
Citing papers per year: 2026: 1.
1 paper.