mlsanityData to 7 Oct 2026

Language Models Can Control Their Own Attention

arXiv 2609.02737 ↗First posted 2 Sep 2026cs.CL cs.AI cs.LGcited by 1

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

Language models spend most of their attention on a small fraction of context, yet they read the entire KV cache to find the few tokens that matter. If the user asks about a previous detail in a 1M-token conversation, global attention layers must scan the full context to generate each token of the reply. A prominent approach mitigates this cost by pre-selecting relevant tokens via lightweight proxy scores, but this extrinsic scoring still incurs O(N) per step. We take an intrinsic approach motivated by the simple question: wouldn't the model already know which parts of the context are relevant? To this end, we introduce Declarative Attention (DA), a protocol that elicits the model to declare where it needs to attend within its chain-of-thought, partitioning generation into three modes: <global> (full context), <focus> (a specific region), and <local> (recent output only). The inference engine parses these declarations like tool calls and skips most of the KV cache read. Under zero-shot evaluation across 15 long-context tasks, DA on off-the-shelf models (Gemma-4-31B, Qwen-3.6-27B) significantly reduces total attended tokens during decoding (52.0%, 31.1%) with modest accuracy drops (1.27pp, 2.75pp) that shrink with model scale. DA unlocks a new axis of sparse attention, with further potential under training-based methods that future work can explore.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 67 of its 90 references that are in the map. First posted on arXiv 2 Sep 2026.

Builds on

67 of 90 references.

Closest ancestors · 12

Same direction, newest first

  1. 2026SpotAttention: Plug-In Block-Sparse Routing for Pretrained Long-Context Transformers22 Jun
  2. 2026Learning to Evict from Key-Value Cache10 Feb
  3. 2024SCBench: A KV Cache-Centric Analysis of Long-Context Methods13 Dec
  4. 2024ClusterKV: Manipulating LLM KV Cache in Semantic Space for Recallable Compression4 Dec
  5. 2024In-context KV-Cache Eviction for LLMs via Attention-Gate15 Oct
  6. 2024InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Management28 Jun
  7. 2024Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference16 Jun
  8. 2024SnapKV: LLM Knows What You are Looking for Before Generation22 Apr
  9. 2023SparQ Attention: Bandwidth-Efficient LLM Inference8 Dec
  10. 2023Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs3 Oct
  11. 2023Efficient Memory Management for Large Language Model Serving with PagedAttention12 Sep
  12. 2023Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Time26 May

Other references · 55

  1. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  3. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  4. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  5. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  6. 2020Longformer: The Long-Document Transformercited by 3,362
  7. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  8. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  9. 2023Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflectioncited by 1,800
  10. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  11. 2019Generating Long Sequences with Sparse Transformerscited by 1,523
  12. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,378
38 more
  1. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,311
  2. 2023MemGPT: Towards LLMs as Operating Systemscited by 1,295
  3. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,270
  4. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,093
  5. 2026DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligencecited by 877
  6. 2025DeepSeek-V3.2: Pushing the Frontier of Open Large Language Modelscited by 755
  7. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 700
  8. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 604
  9. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Servingcited by 595
  10. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 567
  11. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 536
  12. 2022Efficiently Scaling Transformer Inferencecited by 529
  13. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Servecited by 507
  14. 2024FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precisioncited by 507
  15. 2026GLM-5: from Vibe Coding to Agentic Engineeringcited by 502
  16. 2024Gated Delta Networks: Improving Mamba2 with Delta Rulecited by 483
  17. 2025Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attentioncited by 391
  18. 2024LongBench v2: Towards Deeper Understanding and Reasoning on Realistic Long-context Multitaskscited by 367
  19. 2024Quiet-STaR: Language Models Can Teach Themselves to Think Before Speakingcited by 292
  20. 2023Think before you speak: Training Language Models With Pause Tokenscited by 254
  21. 2024Mixture-of-Depths: Dynamically allocating compute in transformer-based language modelscited by 242
  22. 2025Chain of Thought Monitorability: A New and Fragile Opportunity for AI Safetycited by 222
  23. 2023LooGLE: Can Long-Context Language Models Understand Long Contexts?cited by 200
  24. 2024Mooncake: A KVCache-centric Disaggregated Architecture for LLM Servingcited by 193
  25. 2024InfLLM: Training-Free Long-Context Extrapolation for LLMs with an Efficient Context Memorycited by 192
  26. 2026Gemma 4 Technical Reportcited by 187
  27. 2023ZeroSCROLLS: A Zero-Shot Benchmark for Long Text Understandingcited by 175
  28. 2023Future Lens: Anticipating Subsequent Tokens from a Single Hidden Statecited by 116
  29. 2024RetrievalAttention: Accelerating Long-Context LLM Inference via Vector Retrievalcited by 114
  30. 2024MagicPIG: LSH Sampling for Efficient LLM Generationcited by 112
  31. 2026Kimi K3: Open Frontier Intelligencecited by 99
  32. 2023System 2 Attention (is something you might need too)cited by 85
  33. 2023Tell Your Model Where to Attend: Post-hoc Attention Steering for LLMscited by 82
  34. 2024Do language models plan ahead for future tokens?cited by 47
  35. 2024Post-Training Sparse Attention with Double Sparsitycited by 45
  36. 2026MiniMax Sparse Attentioncited by 34
  37. 2024TidalDecode: Fast and Accurate LLM Decoding with Position Persistent Sparse Attentioncited by 28
  38. 2025ThinKV: Thought-Adaptive KV Cache Compression for Efficient Reasoning Modelscited by 21

50 most cited of 55 All 55 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026On-Demand Attention: Language Models Know When to Recallcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Attention Heads of Large Language Models: A Survey
  2. 2025Knowing When to Stop: Efficient Context Processing via Latent Sufficiency Signals
  3. 2025Training-free Context-adaptive Attention for Efficient Long Context Modeling
  4. 2023Attention Sorting Combats Recency Bias In Long Context Language Models
  5. 2026Logit-Contribution Scoring Identifies Non-Literal Retrieval Heads
  6. 2025Don't Take Things Out of Context: Attention Intervention for Enhancing Chain-of-Thought Reasoning in Large Language Models
  7. 2025Every Token Counts: Generalizing 16M Ultra-Long Context in Large Language Models
  8. 2026Rethinking Contextualization by Reinterpreting Attention Head Channels