mlsanityData to 7 Oct 2026

A3: Accelerating Attention Mechanisms in Neural Networks with Approximation

arXiv 2002.10941 ↗First posted 22 Feb 2020cs.DC cs.LGcited by 97

Not on the map. Why?

Abstract (arXiv)

With the increasing computational demands of neural networks, many hardware accelerators for the neural networks have been proposed. Such existing neural network accelerators often focus on popular neural network types such as convolutional neural networks (CNNs) and recurrent neural networks (RNNs); however, not much attention has been paid to attention mechanisms, an emerging neural network primitive that enables neural networks to retrieve most relevant information from a knowledge-base, external memory, or past states. The attention mechanism is widely adopted by many state-of-the-art neural networks for computer vision, natural language processing, and machine translation, and accounts for a large portion of total execution time. We observe today's practice of implementing this mechanism using matrix-vector multiplication is suboptimal as the attention mechanism is semantically a content-based search where a large portion of computations ends up not being used. Based on this observation, we design and architect A3, which accelerates attention mechanisms in neural networks with algorithmic approximation and hardware specialization. Our proposed accelerator achieves multiple orders of magnitude improvement in energy efficiency (performance/watt) as well as substantial speedup over the state-of-the-art conventional hardware.

Key facts. Cited by 97 papers in the map (data through 7 Oct 2026). First posted on arXiv 22 Feb 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 5; 2022: 16; 2023: 13; 2024: 29; 2025: 29; 2026: 5.

97 papers.

Sort:

By area:

  1. 2024QServe: W4A8KV4 Quantization and System Co-design for Efficient LLM Servingcited by 200
  2. 2022A Fast Post-Training Pruning Framework for Transformerscited by 161
  3. 2023Full Stack Optimization of Transformer Inference: a Surveycited by 79
  4. 2022Efficient Methods for Natural Language Processing: A Surveycited by 78
  5. 2024FlightLLM: Efficient Large Language Model Inference with a Complete Mapping Flow on FPGAscited by 60
  6. 2022DFX: A Low-latency Multi-FPGA Appliance for Accelerating Transformer-based Text Generationcited by 46
  7. 2024Mixture of Attention Spans: Optimizing LLM Inference Efficiency with Heterogeneous Sliding-Window Lengthscited by 44
  8. 2022ViTCoD: Vision Transformer Acceleration via Dedicated Algorithm and Accelerator Co-Designcited by 41
  9. 2021FLAT: An Optimized Dataflow for Mitigating Attention Bottleneckscited by 38
  10. 2024Lightweight Deep Learning for Resource-Constrained Environments: A Surveycited by 38
  11. 2023A Survey on Efficient Training of Transformerscited by 37
  12. 2023A Survey of Techniques for Optimizing Transformer Inferencecited by 35
38 more
  1. 2024Duplex: A Device for Large Language Models with Mixture of Experts, Grouped Query Attention, and Continuous Batchingcited by 33
  2. 2022Enable Deep Learning on Mobile Devices: Methods, Systems, and Applicationscited by 32
  3. 2022SimA: Simple Softmax-free Attention for Vision Transformerscited by 32
  4. 2024LLMServingSim: A HW/SW Co-Simulation Infrastructure for LLM Inference Serving at Scalecited by 30
  5. 2022Adaptable Butterfly Accelerator for Attention-based NNs via Hardware and Algorithm Co-designcited by 29
  6. 2022ViTALiTy: Unifying Low-rank and Sparse Approximation for Vision Transformer Acceleration with a Linear Taylor Attentioncited by 29
  7. 2022An Algorithm-Hardware Co-Optimized Framework for Accelerating N:M Sparse Transformerscited by 28
  8. 2021Accelerating Framework of Transformer by Hardware Design and Model Compression Co-Optimizationcited by 27
  9. 2021Energon: Towards Efficient Acceleration of Transformers Using Dynamic Sparse Attentioncited by 26
  10. 2023X-Former: In-Memory Acceleration of Transformerscited by 25
  11. 2023The Efficiency Spectrum of Large Language Models: An Algorithmic Surveycited by 25
  12. 2023AccelTran: A Sparsity-Aware Accelerator for Dynamic Inference with Transformerscited by 24
  13. 2023Chameleon: a Heterogeneous and Disaggregated Accelerator System for Retrieval-Augmented Language Modelscited by 23
  14. 2022A Length Adaptive Algorithm-Hardware Co-design of Transformer on FPGA Through Sparse Attention and Dynamic Pipeliningcited by 21
  15. 2021Transformer Acceleration with Dynamic Sparse Attentioncited by 19
  16. 2025PAPI: Exploiting Dynamic Parallelism in Large Language Model Decoding with a Processing-In-Memory-Enabled Computing Systemcited by 19
  17. 2023Integer or Floating Point? New Outlooks for Low-Bit Quantization on Large Language Modelscited by 18
  18. 2023SwiftTron: An Efficient Hardware Accelerator for Quantized Transformerscited by 17
  19. 2024A Survey on Hardware Accelerators for Large Language Modelscited by 15
  20. 2024ConSmax: Hardware-Friendly Alternative Softmax with Learnable Parameterscited by 14
  21. 2024CHAI: Clustered Head Attention for Efficient LLM Inferencecited by 13
  22. 2022Sparse Attention Acceleration with Synergistic In-Memory Pruning and On-Chip Recomputationcited by 12
  23. 2022SALO: An Efficient Spatial Accelerator Enabling Hybrid Sparse Attention Mechanisms for Long Sequencescited by 10
  24. 2025QUARK: Quantization-Enabled Circuit Sharing for Transformer Acceleration by Exploiting Common Patterns in Nonlinear Operationscited by 10
  25. 2025Stratum: System-Hardware Co-Design with Tiered Monolithic 3D-Stackable DRAM for Efficient MoE Servingcited by 9
  26. 2022Accelerating Attention through Gradient-Based Learned Runtime Pruningcited by 8
  27. 2025QuartDepth: Post-Training Quantization for Real-Time Depth Estimation on the Edgecited by 7
  28. 2023Sparse-DySta: Sparsity-Aware Dynamic and Static Scheduling for Sparse Multi-DNN Workloadscited by 6
  29. 2024BETA: Binarized Energy-Efficient Transformer Accelerator at the Edgecited by 6
  30. 20241.5-Pints Technical Report: Pretraining in Days, Not Months -- Your Language Model Thrives on Quality Datacited by 6
  31. 2024Cross-layer Attention Sharing for Pre-trained Large Language Modelscited by 5
  32. 2025EXION: Exploiting Inter- and Intra-Iteration Output Sparsity for Diffusion Modelscited by 5
  33. 2025Low-Cost FlashAttention with Fused Exponential and Multiplication Hardware Operatorscited by 5
  34. 2025SOLE: Hardware-Software Co-design of Softmax and LayerNorm for Efficient Transformer Inferencecited by 5
  35. 2022GROW: A Row-Stationary Sparse-Dense GEMM Accelerator for Memory-Efficient Graph Convolutional Neural Networkscited by 4
  36. 2024Deploying Foundation Model Powered Agent Services: A Surveycited by 4
  37. 2025Fine-Grained Image-Text Correspondence with Cost Aggregation for Open-Vocabulary Part Segmentationcited by 4
  38. 2025MCBP: A Memory-Compute Efficient LLM Inference Accelerator Leveraging Bit-Slice-enabled Sparsity and Repetitivenesscited by 4

50 most cited of 97 All 97 (JSON)