mlsanityData to 7 Oct 2026

Alleviating Forgetfulness of Linear Attention by Hybrid Sparse Attention and Contextualized Learnable Token Eviction

arXiv 2510.20787 ↗First posted 23 Oct 2025cs.CL cs.LGcited by 3

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

Linear-attention models that compress the entire input sequence into a fixed-size recurrent state offer an efficient alternative to Transformers, but their finite memory induces forgetfulness that harms retrieval-intensive tasks. To mitigate the issue, we explore a series of hybrid models that restore direct access to past tokens. We interleave token mixers with intermediate time and space complexity between linear and full attention, including sparse attention with token eviction, and the query-aware native sparse attention. Particularly, we propose a novel learnable token eviction approach. Combined with sliding-window attention, an end-to-end trainable lightweight CNN aggregates information from both past and future adjacent tokens to adaptively retain a limited set of critical KV-pairs per head, maintaining linear attention's constant time and space complexity. Efficient Triton kernels for the sparse attention mechanisms are provided. Empirical evaluations on retrieval-intensive benchmarks support the effectiveness of our approaches.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 76 of its 89 references that are in the map. First posted on arXiv 23 Oct 2025.

Builds on

76 of 89 references.

Closest ancestors · 13

Same direction, newest first

  1. 2025Inference-Time Hyper-Scaling with KV Cache Compression5 Jun
  2. 2025CAKE: Cascading and Adaptive KV Cache Eviction with Layer Preferences16 Mar
  3. 2024DynamicKV: Task-Aware Adaptive KV Cache Compression for Long Context LLMs19 Dec
  4. 2024In-context KV-Cache Eviction for LLMs via Attention-Gate15 Oct
  5. 2024NACL: A General and Effective KV Cache Eviction Framework for LLMs at Inference Time7 Aug
  6. 2024A2SF: Accumulative Attention Scoring with Forgetting Factor for Token Pruning in Transformer Decoder30 Jul
  7. 2024Ada-KV: Optimizing KV Cache Eviction by Adaptive Budget Allocation for Efficient LLM Inference16 Jul
  8. 2024Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference16 Jun
  9. 2024PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funneling4 Jun
  10. 2024PyramidInfer: Pyramid KV Cache Compression for High-throughput LLM Inference21 May
  11. 2024SnapKV: LLM Knows What You are Looking for Before Generation22 Apr
  12. 2023Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs3 Oct
1 more
  1. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models24 Jun

Other references · 63

  1. 2017Attention Is All You Needcited by 59,932
  2. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
  3. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  4. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  5. 2020Longformer: The Long-Document Transformercited by 3,362
  6. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  7. 2021Efficiently Modeling Long Sequences with Structured State Spacescited by 2,424
  8. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  9. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 1,979
  10. 2020Big Bird: Transformers for Longer Sequencescited by 1,673
  11. 2020Reformer: The Efficient Transformercited by 1,658
  12. 2019Generating Long Sequences with Sparse Transformerscited by 1,523
38 more
  1. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,311
  2. 2024Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Dualitycited by 1,283
  3. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,270
  4. 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,104
  5. 2023RWKV: Reinventing RNNs for the Transformer Eracited by 814
  6. 2023Retentive Network: A Successor to Transformer for Large Language Modelscited by 594
  7. 2023Gated Linear Attention Transformers with Hardware-Efficient Trainingcited by 490
  8. 2024Gated Delta Networks: Improving Mamba2 with Delta Rulecited by 483
  9. 2021Linear Transformers Are Secretly Fast Weight Programmerscited by 471
  10. 2020Efficient Content-Based Sparse Attention with Routing Transformerscited by 469
  11. 2025Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attentioncited by 391
  12. 2024MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attentioncited by 375
  13. 2024Parallelizing Linear Transformers with the Delta Rule over Sequence Lengthcited by 373
  14. 2024Mixture-of-Depths: Dynamically allocating compute in transformer-based language modelscited by 242
  15. 2020SpAtten: Efficient Sparse Attention Architecture with Cascade Token and Head Pruningcited by 228
  16. 2024Repeat After Me: Transformers are Better than State Space Models at Copyingcited by 208
  17. 2024Simple linear attention language models balance the recall-throughput tradeoffcited by 197
  18. 2024InfLLM: Training-Free Long-Context Extrapolation for LLMs with an Efficient Context Memorycited by 192
  19. 2023Zoology: Measuring and Improving Recall in Efficient Language Modelscited by 188
  20. 2025MoBA: Mixture of Block Attention for Long-Context LLMscited by 188
  21. 2024Flex Attention: A Programming Model for Generating Optimized Attention Kernelscited by 180
  22. 2024An Empirical Study of Mamba-based Language Modelscited by 177
  23. 2024Samba: Simple Hybrid State Space Models for Efficient Unlimited Context Language Modelingcited by 167
  24. 2025MiniMax-01: Scaling Foundation Models with Lightning Attentioncited by 157
  25. 2025RWKV-7 "Goose" with Expressive Dynamic State Evolutioncited by 134
  26. 2024HGRN2: Gated Linear RNNs with State Expansioncited by 122
  27. 2024SeerAttention: Learning Intrinsic Sparse Attention in Your LLMscited by 121
  28. 2023LM-Infinite: Zero-Shot Extreme Length Generalization for Large Language Modelscited by 120
  29. 2024Transformers are Multi-State RNNscited by 120
  30. 2025SpargeAttention: Accurate and Training-free Sparse Attention Accelerating Any Model Inferencecited by 113
  31. 2024Loki: Low-rank Keys for Efficient Sparse Attentioncited by 87
  32. 2023CoLT5: Faster Long-Range Transformers with Conditional Computationcited by 84
  33. 2023Dynamic Context Pruning for Efficient and Interpretable Autoregressive Transformerscited by 67
  34. 2024RNNs are not Transformers (Yet): The Key Bottleneck on In-context Retrievalcited by 62
  35. 2024Just read twice: closing the recall gap for recurrent language modelscited by 55
  36. 2024ReMoE: Fully Differentiable Mixture-of-Experts with ReLU Routingcited by 53
  37. 2024ALISA: Accelerating Large Language Model Inference via Sparsity-Aware KV Cachingcited by 48
  38. 2025MoM: Linear Sequence Modeling with Mixture-of-Memoriescited by 44

50 most cited of 63 All 63 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026MonarchRT: Efficient Attention for Real-Time Video Generationcited by 6
  2. 2026MiniCPM-SALA: Hybridizing Sparse and Linear Attention for Efficient Long-Context Modelingcited by 4
  3. 2026A Hippocampus for Linear Attention: An Exact Memory for What the Recurrent State Forgetscited by 4

Similar, not cited

Close in wording, no citation either way

  1. 2026Learning to Forget Attention: Memory Consolidation for Adaptive Compute Reduction
  2. 2025LoLA: Low-Rank Linear Attention With Sparse Caching
  3. 2026RAM-Net: Linear-Time Sequence Modeling with Sparsely Addressable State
  4. 2025Hardware-aligned Hierarchical Sparse Attention for Efficient Long-term Memory Access
  5. 2026Simplified Sparse Attention via Gist Tokens
  6. 2026DART: Decoded Attention over Recurrent States for Efficient Long-Context Sequence Modeling
  7. 2026Augmenting Attention with Exponentially Decaying Memory Improves Query-Aware KV Sparsity
  8. 2026Full Attention Strikes Back: Transferring Full Attention into Sparse within Hundred Training Steps