mlsanityData to 7 Oct 2026

Efficient Sparse Attention needs Adaptive Token Release

arXiv 2407.02328 ↗First posted 2 Jul 2024cs.CLcited by 4

Not on the map. Why?

Abstract (arXiv)

In recent years, Large Language Models (LLMs) have demonstrated remarkable capabilities across a wide array of text-centric tasks. However, their `large' scale introduces significant computational and storage challenges, particularly in managing the key-value states of the transformer, which limits their wider applicability. Therefore, we propose to adaptively release resources from caches and rebuild the necessary key-value states. Particularly, we accomplish this by a lightweight controller module to approximate an ideal top-K sparse attention. This module retains the tokens with the highest top-K attention weights and simultaneously rebuilds the discarded but necessary tokens, which may become essential for future decoding. Comprehensive experiments in natural language generation and modeling reveal that our method is not only competitive with full attention in terms of performance but also achieves a significant throughput improvement of up to 221.8%. The code for replication is available on the https://github.com/WHUIR/ADORE.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 33 of its 53 references that are in the map. First posted on arXiv 2 Jul 2024.

Builds on

33 of 53 references.

References · 33

  1. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  2. 2020Scaling Laws for Neural Language Modelscited by 6,411
  3. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  4. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  5. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  6. 2020Longformer: The Long-Document Transformercited by 3,362
  7. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  8. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  9. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  10. 2020Big Bird: Transformers for Longer Sequencescited by 1,673
  11. 2019Generating Long Sequences with Sparse Transformerscited by 1,523
  12. 2017Get To The Point: Summarization with Pointer-Generator Networkscited by 1,281
21 more
  1. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,085
  2. 2022Token Merging: Your ViT But Fastercited by 995
  3. 2021DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsificationcited by 849
  4. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 825
  5. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 567
  6. 2019SAMSum Corpus: A Human-annotated Dialogue Dataset for Abstractive Summarizationcited by 530
  7. 2023Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Timecited by 428
  8. 2022DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scalecited by 403
  9. 2024LLMLingua-2: Data Distillation for Efficient and Faithful Task-Agnostic Prompt Compressioncited by 343
  10. 2023Deja Vu: Contextual Sparsity for Efficient LLMs at Inference Timecited by 247
  11. 2017Gate-Variants of Gated Recurrent Unit (GRU) Neural Networkscited by 239
  12. 2022Petals: Collaborative Inference and Fine-tuning of Large Modelscited by 84
  13. 2023Predictive Pipelined Decoding: A Compute-Latency Trade-off for Exact LLM Decodingcited by 40
  14. 2023PPT: Token Pruning and Pooling for Efficient Vision Transformerscited by 37
  15. 2022DynaST: Dynamic Sparse Transformer for Exemplar-Guided Image Generationcited by 24
  16. 2021Early Exiting with Ensemble Internal Classifierscited by 19
  17. 2023User Retention-oriented Recommendation with Decision Transformercited by 14
  18. 2023Sparsity-Guided Holistic Explanation for LLMs with Interpretable Inference-Time Interventioncited by 12
  19. 2023Context Compression for Auto-regressive Transformers with Sentinel Tokenscited by 8
  20. 2023NarrowBERT: Accelerating Masked Language Model Pretraining and Inferencecited by 3
  21. 2022Whole Page Unbiased Learning to Rankcited by 2

Built on by

Cited by year

Citing papers per year: 2024: 1; 2025: 3.

4 papers.

  1. 2025Cognitive Memory in Large Language Modelscited by 28
  2. 2025Alleviating Forgetfulness of Linear Attention by Hybrid Sparse Attention and Contextualized Learnable Token Evictioncited by 3
  3. 2024XKV: Personalized KV Cache Memory Reduction for Long-Context LLM Inferencecited by 2
  4. 2025ReSSFormer: A Recursive Sparse Structured Transformer for Scalable and Long-Context Reasoningcited by 0