mlsanityData to 7 Oct 2026

RAT+: Train Dense, Infer Sparse -- Recurrence Augmented Attention for Dilated Inference

arXiv 2602.18196 ↗First posted 20 Feb 2026cs.LGcited by 2

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

Structured dilated attention has an appealing inference-time efficiency knob: it reduces the FLOPs of attention and the KV cache size by a factor of the dilation size D, while preserving long-range connectivity. While prior work studies it by training each configuration from scratch, directly sparsifying a pretrained attention model into a dilated pattern leads to severe accuracy degradation, preventing flexible reuse across inference scenarios. We introduce RAT+, a dense-pretraining architecture that augments attention with full-sequence recurrence and active recurrence learning. A single RAT+ model is pretrained densely once and can then be flexibly switched at inference time to dilated attention (optionally with local windows) or hybrid layer/head compositions, requiring only a short 1B-token resolution adaptation rather than retraining separate sparse models. At 1.5B parameters trained on 100B tokens, RAT+ closely matches dense accuracy at D = 16, and drops by about 2-3 points at D = 64 on commonsense reasoning and LongBench tasks. We further scale to 2.6B and 7.6B parameters and observe even more promising performance (e.g., a 1-point average accuracy loss with a 64x reduction in attention FLOPs and KV cache size). Code is available at https://github.com/wimh966/rat-plus.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 56 of its 63 references that are in the map. First posted on arXiv 20 Feb 2026.

Builds on

56 of 63 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024DuoAttention: Efficient Long-Context LLM Inference with Retrieval and Streaming Heads14 Oct
  2. 2024Ada-KV: Optimizing KV Cache Eviction by Adaptive Budget Allocation for Efficient LLM Inference16 Jul
  3. 2024Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference16 Jun
  4. 2024SnapKV: LLM Knows What You are Looking for Before Generation22 Apr
  5. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models24 Jun

Other references · 51

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  3. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
  4. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  5. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  6. 2020Longformer: The Long-Document Transformercited by 3,362
  7. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  8. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 1,979
  9. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  10. 2020Big Bird: Transformers for Longer Sequencescited by 1,673
  11. 2017Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inferencecited by 1,548
  12. 2019Root Mean Square Layer Normalizationcited by 1,425
38 more
  1. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,378
  2. 2024Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Dualitycited by 1,283
  3. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,270
  4. 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,104
  5. 2023RWKV: Reinventing RNNs for the Transformer Eracited by 814
  6. 2025DeepSeek-V3.2: Pushing the Frontier of Open Large Language Modelscited by 755
  7. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 700
  8. 2019Compressive Transformers for Long-Range Sequence Modellingcited by 699
  9. 2024DeepSeek LLM: Scaling Open-Source Language Models with Longtermismcited by 621
  10. 2023Retentive Network: A Successor to Transformer for Large Language Modelscited by 594
  11. 2023YaRN: Efficient Context Window Extension of Large Language Modelscited by 584
  12. 2024Gated Delta Networks: Improving Mamba2 with Delta Rulecited by 483
  13. 2023Resurrecting Recurrent Neural Networks for Long Sequencescited by 397
  14. 2025Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attentioncited by 391
  15. 2024MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attentioncited by 375
  16. 2023The Impact of Positional Encoding on Length Generalization in Transformerscited by 319
  17. 2025Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Freecited by 266
  18. 2024Griffin: Mixing Gated Linear Recurrences with Local Attention for Efficient Language Modelscited by 232
  19. 2024Retrieval Head Mechanistically Explains Long-Context Factualitycited by 219
  20. 2023LongNet: Scaling Transformers to 1,000,000,000 Tokenscited by 201
  21. 2024InfLLM: Training-Free Long-Context Extrapolation for LLMs with an Efficient Context Memorycited by 192
  22. 2022Transformer Quality in Linear Timecited by 189
  23. 2025MoBA: Mixture of Block Attention for Long-Context LLMscited by 188
  24. 2024Flex Attention: A Programming Model for Generating Optimized Attention Kernelscited by 180
  25. 2018Online normalizer calculation for softmaxcited by 154
  26. 2022Transformer Language Models without Positional Encodings Still Learn Positional Informationcited by 148
  27. 2025FlexPrefill: A Context-Aware Sparse Attention Mechanism for Efficient Long-Sequence Inferencecited by 121
  28. 2023LM-Infinite: Zero-Shot Extreme Length Generalization for Large Language Modelscited by 120
  29. 2024Transformers are Multi-State RNNscited by 120
  30. 2025SpargeAttention: Accurate and Training-free Sparse Attention Accelerating Any Model Inferencecited by 113
  31. 2021A Practical Survey on Faster and Lighter Transformerscited by 73
  32. 2024Gated Slot Attention for Efficient Linear-Time Sequence Modelingcited by 73
  33. 2022Dilated Neighborhood Attention Transformercited by 64
  34. 2025Command A: An Enterprise-Ready Large Language Modelcited by 57
  35. 2025Twilight: Adaptive Attention Sparsity with Hierarchical Top-p Pruningcited by 48
  36. 2025Rope to Nope and Back Again: A New Hybrid Attention Strategycited by 46
  37. 2016Recurrent Batch Normalizationcited by 35
  38. 2017Effective Use of Dilated Convolutions for Segmenting Small Object Instances in Remote Sensing Imagerycited by 20

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Augmenting Attention with Exponentially Decaying Memory Improves Query-Aware KV Sparsitycited by 0
  2. 2026KV-Kaizen: Learning Context-Adaptive Cache Compression Choicescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Attention Routing Stabilizes Early: Working-Set Inference for Recurrent Language Models
  2. 2025Kascade: A Practical Sparse Attention Method for Long-Context LLM Inference
  3. 2025Rectified Sparse Attention
  4. 2025Tactic: Adaptive Sparse Attention with Clustering and Distribution Fitting for Long-Context LLMs
  5. 2026AB-Sparse: Sparse Attention with Adaptive Block Size for Accurate and Efficient Long-Context Inference
  6. 2026Double-P: Hierarchical Top-P Sparse Attention for Long-Context LLMs
  7. 2025Less Is More: Fast and Accurate Reasoning with Cross-Head Unified Sparse Attention
  8. 2026Learning to Forget Attention: Memory Consolidation for Adaptive Compute Reduction