mlsanityData to 7 Oct 2026

SpAtten: Efficient Sparse Attention Architecture with Cascade Token and Head Pruning

arXiv 2012.09852 ↗First posted 17 Dec 2020cs.AR cs.AI cs.CLcited by 228

Not on the map. Why?

Cited by 72 in Efficient LLM inference and serving, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

The attention mechanism is becoming increasingly popular in Natural Language Processing (NLP) applications, showing superior performance than convolutional and recurrent architectures. However, attention becomes the compution bottleneck because of its quadratic computational complexity to input length, complicated data movement and low arithmetic intensity. Moreover, existing NN accelerators mainly focus on optimizing convolutional or recurrent models, and cannot efficiently support attention. In this paper, we present SpAtten, an efficient algorithm-architecture co-design that leverages token sparsity, head sparsity, and quantization opportunities to reduce the attention computation and memory access. Inspired by the high redundancy of human languages, we propose the novel cascade token pruning to prune away unimportant tokens in the sentence. We also propose cascade head pruning to remove unessential heads. Cascade pruning is fundamentally different from weight pruning since there is no trainable weight in the attention mechanism, and the pruned tokens and heads are selected on the fly. To efficiently support them on hardware, we design a novel top-k engine to rank token and head importance scores with high throughput. Furthermore, we propose progressive quantization that first fetches MSBs only and performs the computation; if the confidence is low, it fetches LSBs and recomputes the attention outputs, trading computation for memory reduction. Extensive experiments on 30 benchmarks show that, on average, SpAtten reduces DRAM access by 10.0x with no accuracy loss, and achieves 1.6x, 3.0x, 162x, 347x speedup, and 1,4x, 3.2x, 1193x, 4059x energy savings over A3 accelerator, MNNFast accelerator, TITAN Xp GPU, Xeon CPU, respectively.

Key facts. Cited by 228 papers in the map (data through 7 Oct 2026). First posted on arXiv 17 Dec 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 16; 2022: 28; 2023: 37; 2024: 61; 2025: 57; 2026: 29.

228 papers.

Sort:

By area:

  1. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  2. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,338
  3. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,234
  4. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Modelscited by 805
  5. 2024MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attentioncited by 375
  6. 2023SpecInfer: Accelerating Generative Large Language Model Serving with Tree-based Speculative Inference and Verificationcited by 359
  7. 2025Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Freecited by 266
  8. 2024QServe: W4A8KV4 Quantization and System Co-design for Efficient LLM Servingcited by 200
  9. 2024A Survey on Efficient Inference for Large Language Modelscited by 197
  10. 2023LLM in a flash: Efficient Large Language Model Inference with Limited Memorycited by 170
  11. 2021SPViT: Enabling Faster Vision Transformers via Soft Token Pruningcited by 165
  12. 2022A Fast Post-Training Pruning Framework for Transformerscited by 161
38 more
  1. 2023REST: Retrieval-Based Speculative Decodingcited by 148
  2. 2021Learned Token Pruning for Transformerscited by 138
  3. 2024Keyformer: KV Cache Reduction through Key Tokens Selection for Efficient Generative Inferencecited by 124
  4. 2024SeerAttention: Learning Intrinsic Sparse Attention in Your LLMscited by 121
  5. 2024Dynamic Memory Compression: Retrofitting LLMs for Accelerated Inferencecited by 115
  6. 2021QuantumNAS: Noise-Adaptive Search for Robust Quantum Circuitscited by 97
  7. 2023S3: Increasing GPU Utilization during Generative Inference for Higher Throughputcited by 81
  8. 2022Compression of Generative Pre-trained Language Models via Quantizationcited by 79
  9. 2023Full Stack Optimization of Transformer Inference: a Surveycited by 79
  10. 2022Efficient Methods for Natural Language Processing: A Surveycited by 78
  11. 2024Model Compression and Efficient Inference for Large Language Models: A Surveycited by 62
  12. 2023Zero-TPrune: Zero-Shot Token Pruning through Leveraging of the Attention Graph in Pre-Trained Transformerscited by 61
  13. 2023SparseViT: Revisiting Activation Sparsity for Efficient High-Resolution Vision Transformercited by 60
  14. 2024FlightLLM: Efficient Large Language Model Inference with a Complete Mapping Flow on FPGAscited by 60
  15. 2021Keyword Transformer: A Self-Attention Model for Keyword Spottingcited by 59
  16. 2024Fast On-device LLM Inference with NPUscited by 58
  17. 2024MoE-Pruner: Pruning Mixture-of-Experts Large Language Model using the Hints from Its Routercited by 58
  18. 2025Token Reduction Should Go Beyond Efficiency in Generative Models -- From Vision, Language to Multimodalitycited by 49
  19. 2024ALISA: Accelerating Large Language Model Inference via Sparsity-Aware KV Cachingcited by 48
  20. 2022DFX: A Low-latency Multi-FPGA Appliance for Accelerating Transformer-based Text Generationcited by 46
  21. 2023LLMCad: Fast and Scalable On-device Large Language Model Inferencecited by 45
  22. 2023CrossGET: Cross-Guided Ensemble of Tokens for Accelerating Vision-Language Transformerscited by 44
  23. 2024Mixture of Attention Spans: Optimizing LLM Inference Efficiency with Heterogeneous Sliding-Window Lengthscited by 44
  24. 2024AIM: Adaptive Inference of Multi-Modal LLMs via Token Merging and Pruningcited by 43
  25. 2022ANT: Exploiting Adaptive Numerical Data Type for Low-bit Deep Neural Network Quantizationcited by 42
  26. 2023Discrete Prompt Compression with Reinforcement Learningcited by 42
  27. 2022ViTCoD: Vision Transformer Acceleration via Dedicated Algorithm and Accelerator Co-Designcited by 41
  28. 2021FLAT: An Optimized Dataflow for Mitigating Attention Bottleneckscited by 38
  29. 2024Lightweight Deep Learning for Resource-Constrained Environments: A Surveycited by 38
  30. 2025DPad: Efficient Diffusion Language Models with Suffix Dropoutcited by 38
  31. 2024Large Language Model Inference Acceleration: A Comprehensive Hardware Perspectivecited by 36
  32. 2023A Survey of Techniques for Optimizing Transformer Inferencecited by 35
  33. 2024Duplex: A Device for Large Language Models with Mixture of Experts, Grouped Query Attention, and Continuous Batchingcited by 33
  34. 2022Enable Deep Learning on Mobile Devices: Methods, Systems, and Applicationscited by 32
  35. 2024Block Transformer: Global-to-Local Language Modeling for Fast Inferencecited by 32
  36. 2023What Matters In The Structured Pruning of Generative Language Models?cited by 30
  37. 2024LLMServingSim: A HW/SW Co-Simulation Infrastructure for LLM Inference Serving at Scalecited by 30
  38. 2022Adaptable Butterfly Accelerator for Attention-based NNs via Hardware and Algorithm Co-designcited by 29

50 most cited of 228 All 228 (JSON)