mlsanityData to 14 Oct 2026

ART: Attention Run-time Termination for Efficient Large Language Model Decoding

arXiv 2606.00024 ↗First posted 15 Apr 2026cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

Long-context decoding in Large Language Models (LLMs) is constrained by the cost of accessing and processing the Key-Value (KV) cache. Despite evidence that attention outputs depend jointly on keys and values, most existing KV management methods rely on key-only pruning, since incorporating values incurs prohibitive overhead. In this paper, we propose Attention Run-time Termination (ART), a lightweight run-time mechanism that tracks accumulated attention outputs during kernel execution and terminates subsequent KV block accesses once further contributions become negligible. Rather than replacing KV selection, ART dynamically terminates redundant KV traversal on top of existing dense or sparse attention policies. We introduce a stability-based criterion that monitors both magnitude and directional changes of intermediate attention outputs and provideds a theoretical characterization of the resulting truncation error. Experiments on the LongBench and RULER Needle-in-a-Haystack tasks show that ART increases the generation throughput of existing KV-cache methods by up to 20%, without compromising the result quality.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 22 of its 23 references that are in the map. First posted on arXiv 15 Apr 2026.

Builds on

22 of 23 references.

References · 22

  1. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  2. 2025Qwen3 Technical Reportcited by 8,820
  3. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  4. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  5. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  6. 2023Mistral 7Bcited by 3,164
  7. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  8. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  9. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,417
  10. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,346
  11. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,141
  12. 2024SnapKV: LLM Knows What You are Looking for Before Generationcited by 880
10 more
  1. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Modelscited by 832
  2. 2024Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inferencecited by 501
  3. 2023Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Timecited by 439
  4. 2024PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funnelingcited by 432
  5. 2023LM-Infinite: Zero-Shot Extreme Length Generalization for Large Language Modelscited by 124
  6. 2024Transformers are Multi-State RNNscited by 120
  7. 2024Attention Score is not All You Need for Token Importance Indicator in KV Cache Reduction: Value Also Matterscited by 73
  8. 2025ChunkKV: Semantic-Preserving KV Cache Compression for Efficient Long-Context LLM Inferencecited by 60
  9. 2025TokenButler: Token Importance is Predictablecited by 12
  10. 2025OBCache: Optimal Brain KV Cache Pruning for Efficient Long-Context LLM Inferencecited by 10

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Sliding-window beats linear attention
  2. 2024A Simple and Effective L2 Norm-Based Strategy for KV Cache Compression
  3. 2026CateKV: On Sequential Consistency for Long-Context LLM Inference Acceleration
  4. 2025KVSink: Understanding and Enhancing the Preservation of Attention Sinks in KV Cache Quantization for LLMs
  5. 2025Progressive Sparse Attention: Algorithm and System Co-design for Efficient Attention in LLM Serving
  6. 2026Moment-KV: Momentum-Based Decode-Time KV Cache Compression for Long Generation
  7. 2024Beyond KV Caching: Shared Attention for Efficient LLMs
  8. 2026PuzzleKV: Page-Wise Low-Rank Decomposition for KV Cache Compression