mlsanityData to 7 Oct 2026

TokenSelect: Efficient Long-Context Inference and Length Extrapolation for LLMs via Dynamic Token-Level KV Cache Selection

arXiv 2411.02886 ↗First posted 5 Nov 2024cs.CL cs.AI cs.LGcited by 38

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

Rapid advances in Large Language Models (LLMs) have spurred demand for processing extended context sequences in contemporary applications. However, this progress faces two challenges: performance degradation due to sequence lengths out-of-distribution, and excessively long inference times caused by the quadratic computational complexity of attention. These issues limit LLMs in long-context scenarios. In this paper, we propose Dynamic Token-Level KV Cache Selection (TokenSelect), a training-free method for efficient and accurate long-context inference. TokenSelect builds upon the observation of non-contiguous attention sparsity, using QK dot products to measure per-head KV Cache criticality at token-level. By per-head soft voting mechanism, TokenSelect selectively involves a few critical KV cache tokens in attention calculation without sacrificing accuracy. To further accelerate TokenSelect, we design the Selection Cache based on observations of consecutive Query similarity and implemented the efficient Paged Dot Product Kernel, significantly reducing the selection overhead. A comprehensive evaluation of TokenSelect demonstrates up to 23.84× speedup in attention computation and up to 2.28× acceleration in end-to-end latency, while providing superior performance compared to state-of-the-art long-context inference methods.

Key facts. Cited by 38 papers in the map (data through 7 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 46 of its 74 references that are in the map. First posted on arXiv 5 Nov 2024.

Builds on

46 of 74 references.

Closest ancestors · 12

Same direction, newest first

  1. 2024ShadowKV: KV Cache in Shadows for High-Throughput Long-Context LLM Inference28 Oct
  2. 2024RazorAttention: Efficient KV Cache Compression Through Retrieval Heads22 Jul
  3. 2024Model Tells You Where to Merge: Adaptive KV Cache Merging for LLMs on Long-Context Tasks11 Jul
  4. 2024InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Management28 Jun
  5. 2024Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference16 Jun
  6. 2024SnapKV: LLM Knows What You are Looking for Before Generation22 Apr
  7. 2024GEAR: An Efficient KV Cache Compression Recipe for Near-Lossless Generative Inference of LLM8 Mar
  8. 2024No Token Left Behind: Reliable KV Cache Compression via Importance-Aware Mixed Precision Quantization28 Feb
  9. 2023SparQ Attention: Bandwidth-Efficient LLM Inference8 Dec
  10. 2023Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs3 Oct
  11. 2023Efficient Memory Management for Large Language Model Serving with PagedAttention12 Sep
  12. 2023Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Time26 May

Other references · 34

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  5. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  6. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  7. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  8. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  9. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  10. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  11. 2020Big Bird: Transformers for Longer Sequencescited by 1,673
  12. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,378
22 more
  1. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,270
  2. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,107
  3. 2024ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Toolscited by 1,096
  4. 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 721
  5. 2019Compressive Transformers for Long-Range Sequence Modellingcited by 699
  6. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 604
  7. 2024MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attentioncited by 375
  8. 2024∞Bench: Extending Long Context Evaluation Beyond 100K Tokenscited by 360
  9. 2023DeepSpeed Ulysses: System Optimizations for Enabling Training of Extreme Long Sequence Transformer Modelscited by 239
  10. 2024Leave No Context Behind: Efficient Infinite Context Transformers with Infini-attentioncited by 194
  11. 2024InfLLM: Training-Free Long-Context Extrapolation for LLMs with an Efficient Context Memorycited by 192
  12. 2023LM-Infinite: Zero-Shot Extreme Length Generalization for Large Language Modelscited by 120
  13. 2024RetrievalAttention: Accelerating Long-Context LLM Inference via Vector Retrievalcited by 114
  14. 2019Multi-passage BERT: A Globally Normalized BERT Model for Open-domain Question Answeringcited by 96
  15. 2024ZipCache: Accurate and Efficient KV Cache Quantization with Salient Token Identificationcited by 86
  16. 2024Plan-on-Graph: Self-Correcting Adaptive Planning of Large Language Model on Knowledge Graphscited by 77
  17. 2023Advancing Transformer Architecture in Long-Context Large Language Models: A Comprehensive Surveycited by 71
  18. 2024Post-Training Sparse Attention with Double Sparsitycited by 45
  19. 2023Giraffe: Adventures in Expanding Context Lengths in LLMscited by 32
  20. 2023Length Extrapolation of Transformers: A Survey from the Perspective of Positional Encodingcited by 32
  21. 2024Untie the Knots: An Efficient Data Augmentation Strategy for Long-Context Pre-Training in Language Modelscited by 9
  22. 2024Structure-Enhanced Protein Instruction Tuning: Towards General-Purpose Protein Understanding with LLMscited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 20; 2026: 18.

38 papers.

Sort:

By area:

  1. 2025Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attentioncited by 391
  2. 2025On Memory Construction and Retrieval for Personalized Conversational Agentscited by 101
  3. 2025Token Reduction Should Go Beyond Efficiency in Generative Models -- From Vision, Language to Multimodalitycited by 49
  4. 2025Rethinking Memory in LLM based Agents: Representations, Operations, and Emerging Topicscited by 36
  5. 2025LEMMA: Learning from Errors for MatheMatical Advancement in LLMscited by 20
  6. 2025RetroInfer: A Vector Storage Engine for Scalable Long-Context LLM Inferencecited by 13
  7. 2025TokenButler: Token Importance is Predictablecited by 11
  8. 2025Beyond Exponential Decay: Rethinking Error Accumulation in Large Language Modelscited by 11
  9. 2025AssoMem: Scalable Memory QA with Multi-Signal Associative Retrievalcited by 10
  10. 2025ProxyAttn: Guided Sparse Attention via Representative Headscited by 9
  11. 2025Accelerating Large-Scale Reasoning Model Inference with Sparse Self-Speculative Decodingcited by 9
  12. 2025UniGist: Towards General and Hardware-aligned Sequence-level Long Context Compressioncited by 8
26 more
  1. 2025ContextPilot: Fast Long-Context Inference via Context Reusecited by 8
  2. 2025Progressive Sparse Attention: Algorithm and System Co-design for Efficient Attention in LLM Servingcited by 7
  3. 2025SparseServe: Unlocking Parallelism for Dynamic Sparse Attention in Long-Context LLM Servingcited by 5
  4. 2025KVSwap: Disk-aware KV Cache Offloading for Long-Context On-device Inferencecited by 5
  5. 2026Vegas: Self-Speculative Decoding with Verification-Guided Sparse Attentioncited by 5
  6. 2025Knowing When to Stop: Efficient Context Processing via Latent Sufficiency Signalscited by 4
  7. 2025Retrospective Sparse Attention for Efficient Long-Context Generationcited by 4
  8. 2026Unifying Sparse Attention with Hierarchical Memory for Scalable Long-Context LLM Servingcited by 4
  9. 2026The Model Knows, the Decoder Finds: Future Value Guided Particle Power Samplingcited by 3
  10. 2026Can Language Models Actually Retrieve In-Context? Drowning in Documents at Million Token Scalecited by 2
  11. 2026Towards Efficient Large Language Model Serving: A Survey on System-Aware KV Cache Optimizationcited by 2
  12. 2026LOCKS: Page-Local Compact Key Summaries for Efficient Long-Context Decodingcited by 1
  13. 2026CoSA: Accelerating Long-Context Inference via Proxy-Kernel Co-Designed Sparse Attentioncited by 1
  14. 2025PatternKV: Flattening KV Representation Expands Quantization Headroomcited by 0
  15. 2025CTkvr: KV Cache Retrieval for Long-Context LLMs via Centroid then Token Indexingcited by 0
  16. 2026Attention-Mass Condensation for Sparse Decodingcited by 0
  17. 2026Near-Oracle KV Selection via Pre-hoc Sparsity for Long-Context Inferencecited by 0
  18. 2026KVSlimmer: Theoretical Insights and Practical Optimizations for Asymmetric KV Mergingcited by 0
  19. 2026UT-ACA: Uncertainty-Triggered Adaptive Context Allocation for Long-Context Inferencecited by 0
  20. 2026Compute Where it Counts: Self Optimizing Language Modelscited by 0
  21. 2026Recall Before You Rank: Similarity-Guided Top-K Reuse for Efficient Long-Context Attentioncited by 0
  22. 2026TALSC: Timeliness-Aware Large-Small VLM Collaboration for Infrastructure-Assisted Autonomous Drivingcited by 0
  23. 2026Against Political Polarization: A Unified Framework for Tracing Evolving Political Ideologies on Social Mediacited by 0
  24. 2026HeadWiseKV: Budgeted Per-Head Cache Residency for Hybrid Long-Context Language Modelscited by 0
  25. 2026DeepEdu-v1: Efficient and Scalable Agentic LLMs for Vietnamese Educationcited by 0
  26. 2026CommunityKV: Efficient Long-Context Decoding via Graph Partitioningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025LLMs Know What to Drop: Self-Attention Guided KV Cache Eviction for Efficient Long-Context Inference
  2. 2024Compressing KV Cache for Long-Context LLM Inference with Inter-Layer Attention Similarity
  3. 2026Reformulating KV Cache Eviction Problem for Long-Context LLM Inference
  4. 2024In-context KV-Cache Eviction for LLMs via Attention-Gate
  5. 2026ARKV: Adaptive and Resource-Efficient KV Cache Management under Limited Memory Budget for Long-Context Inference in LLMs
  6. 2025KVCompose: Efficient Structured KV Cache Compression with Composite Tokens
  7. 2024DuoAttention: Efficient Long-Context LLM Inference with Retrieval and Streaming Heads
  8. 2026CateKV: On Sequential Consistency for Long-Context LLM Inference Acceleration