mlsanityData to 7 Oct 2026

Token-Picker: Accelerating Attention in Text Generation with Minimized Memory Transfer via Probability Estimation

arXiv 2407.15131 ↗First posted 21 Jul 2024cs.AR cs.LGcited by 3

Not on the map. Why?

Abstract (arXiv)

The attention mechanism in text generation is memory-bounded due to its sequential characteristics. Therefore, off-chip memory accesses should be minimized for faster execution. Although previous methods addressed this by pruning unimportant tokens, they fall short in selectively removing tokens with near-zero attention probabilities in each instance. Our method estimates the probability before the softmax function, effectively removing low probability tokens and achieving an 12.1x pruning ratio without fine-tuning. Additionally, we present a hardware design supporting seamless on-demand off-chip access. Our approach shows 2.6x reduced memory accesses, leading to an average 2.3x speedup and a 2.4x energy efficiency.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 9 of its 16 references that are in the map. First posted on arXiv 21 Jul 2024.

Builds on

9 of 16 references.

References · 9

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  5. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  6. 2016Pointer Sentinel Mixture Modelscited by 2,992
  7. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  8. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,201
  9. 2020SpAtten: Efficient Sparse Attention Architecture with Cascade Token and Head Pruningcited by 228

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 2; 2026: 1.

3 papers.

  1. 2025BitStopper: An Efficient Transformer Attention Accelerator via Stage-fusion and Early Terminationcited by 1
  2. 2025LAPA: Log-Domain Prediction-Driven Dynamic Sparsity Accelerator for Transformer Modelcited by 1
  3. 2026HiKV: Hierarchical Importance-Aware KV Cache with Hardware Acceleration for LLM Decodingcited by 0