mlsanityData to 7 Oct 2026

Elastic Attention: Test-time Adaptive Sparsity Ratios for Efficient Transformers

arXiv 2601.17367 ↗First posted 24 Jan 2026cs.CL cs.AIcited by 4

Not on the map. Why?

Abstract (arXiv)

The quadratic complexity of standard attention mechanisms poses a significant scalability bottleneck for large language models (LLMs) in long-context scenarios. While hybrid attention strategies that combine sparse and full attention within a single model offer a viable solution, they typically employ static computation ratios (i.e., fixed proportions of sparse versus full attention) and fail to adapt to the varying sparsity sensitivities of downstream tasks during inference. To address this issue, we propose Elastic Attention, which allows the model to dynamically adjust its overall sparsity based on the input. This is achieved by integrating a lightweight Attention Router into the existing pretrained model, which dynamically assigns each attention head to different computation modes. Within only 12 hours of training on 8xA800 GPUs, our method enables models to achieve both strong performance and efficient inference. Experiments across three long-context benchmarks on widely-used LLMs demonstrate the superiority of our method.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 41 of its 49 references that are in the map. First posted on arXiv 24 Jan 2026.

Builds on

41 of 49 references.

References · 41

  1. 2017Attention Is All You Needcited by 59,932
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  4. 2020Longformer: The Long-Document Transformercited by 3,362
  5. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  6. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  7. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,031
  8. 2020Big Bird: Transformers for Longer Sequencescited by 1,673
  9. 2019Generating Long Sequences with Sparse Transformerscited by 1,523
  10. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,378
  11. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,308
  12. 2024Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Dualitycited by 1,283
29 more
  1. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,270
  2. 2021MuSiQue: Multihop Questions via Single-hop Question Compositioncited by 893
  3. 2024SnapKV: LLM Knows What You are Looking for Before Generationcited by 848
  4. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Modelscited by 805
  5. 2023Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Timecited by 428
  6. 2025Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attentioncited by 391
  7. 2024MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attentioncited by 375
  8. 2024LongBench v2: Towards Deeper Understanding and Reasoning on Realistic Long-context Multitaskscited by 367
  9. 2021Efficient Attentions for Long Document Summarizationcited by 342
  10. 2024DuoAttention: Efficient Long-Context LLM Inference with Retrieval and Streaming Headscited by 283
  11. 2024Retrieval Head Mechanistically Explains Long-Context Factualitycited by 219
  12. 2025MoBA: Mixture of Block Attention for Long-Context LLMscited by 188
  13. 2024Samba: Simple Hybrid State Space Models for Efficient Unlimited Context Language Modelingcited by 167
  14. 2025XAttention: Block Sparse Attention with Antidiagonal Scoringcited by 131
  15. 2024SeerAttention: Learning Intrinsic Sparse Attention in Your LLMscited by 121
  16. 2025FlexPrefill: A Context-Aware Sparse Attention Mechanism for Efficient Long-Sequence Inferencecited by 121
  17. 2025A Survey of Context Engineering for Large Language Modelscited by 120
  18. 2024Zamba: A Compact 7B SSM Hybrid Modelcited by 118
  19. 2025A Comprehensive Survey on Long Context Language Modelingcited by 113
  20. 2024ChatQA 2: Bridging the Gap to Proprietary LLMs in Long Context and RAG Capabilitiescited by 52
  21. 2025InfLLM-V2: Dense-Sparse Switchable Attention for Seamless Short-to-Long Adaptationcited by 31
  22. 2025Thus Spake Long-Context Large Language Modelcited by 12
  23. 2025Cache Me If You Can: How Many KVs Do You Need for Effective Long-Context LMs?cited by 10
  24. 2025aiXcoder-7B-v2: Training LLMs to Fully Utilize the Long Context in Repository-level Code Completioncited by 8
  25. 2025Efficient Context Scaling with LongCat ZigZag Attentioncited by 6
  26. 2025LOOM-Scope: a comprehensive and efficient LOng-cOntext Model evaluation frameworkcited by 5
  27. 2025Accelerating Prefilling for Long-Context LLMs via Sparse Pattern Sharingcited by 4
  28. 2025Accelerating Prefilling via Decoding-time Contribution Sparsitycited by 4
  29. 2025SALE : Low-bit Estimation for Efficient Sparse Attention in Long-context LLM Prefillingcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 4.

4 papers.

  1. 2026HydraHead: From Head-Level Functional Heterogeneity to Specialized Attention Hybridizationcited by 3
  2. 2026CoSA: Accelerating Long-Context Inference via Proxy-Kernel Co-Designed Sparse Attentioncited by 1
  3. 2026Self-Tuning Sparse Attention: Multi-Fidelity Hyperparameter Optimization for Transformer Accelerationcited by 0
  4. 2026Flux Attention: Context-Aware Hybrid Attention for Efficient LLMs Inferencecited by 0