mlsanityData to 7 Oct 2026

Sigmoid Attention as a Better Substrate for Learned KV Cache Eviction

arXiv 2608.23296 ↗First posted 24 Aug 2026cs.LG cs.AIcited by 0

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

Learned KV-cache eviction often faces a soft-to-hard mismatch: during training, differentiable gates typically attenuate token contributions, whereas inference saves memory only when KV entries are physically removed. We ask whether the attention substrate affects this soft-to-hard transition. Using GPT-2-scale Transformers trained on OpenWebText, we run a controlled 2×2×2 comparison over attention type, learned gating, and positional encoding. Although sigmoid attention is worse as a dense language model, learned hard eviction changes the useful operating points: sigmoid-gated models delete KV entries with negligible PPL change relative to their own no-eviction references. Under a matched live-cache protocol on the same dense backbones, learned sigmoid gates obtain lower PPL than our H2O and KeyDiff implementations, whereas softmax gates do not uniformly beat these post-hoc methods. The results suggest that attention normalization can substantially affect whether a training-time soft gate transfers cleanly to hard KV deletion.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 16 of its 19 references that are in the map. First posted on arXiv 24 Aug 2026.

Builds on

16 of 19 references.

Closest ancestors · 8

Same direction, newest first

  1. 2026Fast KVzip: Efficient and Accurate LLM Inference with Gated KV Eviction25 Jan
  2. 2025Cache What Lasts: Token Retention for Memory-Bounded KV Cache in LLMs3 Dec
  3. 2025KeyDiff: Key Similarity-Based KV Cache Eviction for Long-Context LLM Inference in Resource-Constrained Environments21 Apr
  4. 2024In-context KV-Cache Eviction for LLMs via Attention-Gate15 Oct
  5. 2024DuoAttention: Efficient Long-Context LLM Inference with Retrieval and Streaming Heads14 Oct
  6. 2024When Attention Sink Emerges in Language Models: An Empirical View14 Oct
  7. 2024D2O: Dynamic Discriminative Operations for Efficient Long-Context Inference of Large Language Models18 Jun
  8. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models24 Jun

Other references · 8

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  3. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  4. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  5. 2019Root Mean Square Layer Normalizationcited by 1,425
  6. 2024Dynamic Memory Compression: Retrofitting LLMs for Accelerated Inferencecited by 115
  7. 2024Theory, Analysis, and Best Practices for Sigmoid Self-Attentioncited by 62
  8. 2025KV-Distill: Nearly Lossless Learnable Context Compression for LLMscited by 19

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026ValueDiff: Value-Geometric KV Cache Eviction for Sink-Suppressed LLMs
  2. 2026Random Attention: Rethinking KV Cache Eviction for Efficient Reasoning
  3. 2025G-KV: Decoding-Time KV Cache Eviction with Global Attention
  4. 2026Jacap: Robust KV Cache Eviction via Jacobian-Based Nonlinear Information Capacity Preservation
  5. 2026Behavior-Preserving KV Cache Compression
  6. 2026KVpop -- Key-Value Cache Compression with Predictive Online Pruning
  7. 2025CAOTE: KV Cache Selection for LLMs via Attention Output Error-Based Token Eviction
  8. 2025KVSink: Understanding and Enhancing the Preservation of Attention Sinks in KV Cache Quantization for LLMs