mlsanityData to 7 Oct 2026

Efficient Softmax Approximation for Deep Neural Networks with Attention Mechanism

arXiv 2111.10770 ↗First posted 21 Nov 2021cs.LG cs.AIcited by 9

Not on the map. Why?

Abstract (arXiv)

There has been a rapid advance of custom hardware (HW) for accelerating the inference speed of deep neural networks (DNNs). Previously, the softmax layer was not a main concern of DNN accelerating HW, because its portion is relatively small in multi-layer perceptron or convolutional neural networks. However, as the attention mechanisms are widely used in various modern DNNs, a cost-efficient implementation of softmax layer is becoming very important. In this paper, we propose two methods to approximate softmax computation, which are based on the usage of LookUp Tables (LUTs). The required size of LUT is quite small (about 700 Bytes) because ranges of numerators and denominators of softmax are stable if normalization is applied to the input. We have validated the proposed technique over different AI tasks (object detection, machine translation, sentiment analysis, and semantic equivalence) and DNN models (DETR, Transformer, BERT) by a variety of benchmarks (COCO17, WMT14, WMT17, GLUE). We showed that 8-bit approximation allows to obtain acceptable accuracy loss below 1.0%.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Builds on 25 of its 36 references that are in the map. First posted on arXiv 21 Nov 2021.

Builds on

25 of 36 references.

References · 25

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  5. 2020End-to-End Object Detection with Transformerscited by 7,384
  6. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  7. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  8. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  9. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  10. 2019Q-BERT: Hessian Based Ultra Low Precision Quantization of BERTcited by 355
  11. 2019Q8BERT: Quantized 8Bit BERTcited by 296
  12. 2019Deep Learning for Symbolic Mathematicscited by 266
13 more
  1. 2017OpenNMT: Open-source Toolkit for Neural Machine Translationcited by 251
  2. 2018Streaming End-to-end Speech Recognition For Mobile Devicescited by 230
  3. 2018Bottom-Up Abstractive Summarizationcited by 177
  4. 2020A Streaming On-Device End-to-End Model Surpassing Server-Side Conventional Model Quality and Latencycited by 99
  5. 2020Molecule Attention Transformercited by 92
  6. 2019Machine Learning at the Network Edge: A Surveycited by 83
  7. 2018AI Benchmark: Running Deep Neural Networks on Android Smartphonescited by 75
  8. 2019Efficient 8-Bit Quantization of Transformer Neural Machine Language Translation Modelcited by 58
  9. 2021Softermax: Hardware/Software Co-Design of an Efficient Softmax for Transformerscited by 52
  10. 2020Towards Fast and Accurate Streaming End-to-End ASRcited by 47
  11. 2019Neural Network Inference on Mobile SoCscited by 17
  12. 2019OpenEI: An Open Framework for Edge Intelligencecited by 12
  13. 2019Softmax Optimizations for Intel Xeon Processor-based Platformscited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 0; 2024: 3; 2025: 3; 2026: 2.

9 papers.

  1. 2022SimA: Simple Softmax-free Attention for Vision Transformerscited by 32
  2. 2024Integer-only Quantized Transformers for Embedded FPGA-based Time-series Forecasting in AIoTcited by 6
  3. 2025Mixed-Precision Quantization for Language Models: Techniques and Prospectscited by 4
  4. 2024EXAQ: Exponent Aware Quantization For LLMs Accelerationcited by 3
  5. 2025Optimizing Neural Networks with Learnable Non-Linear Activation Functions via Lookup-Based FPGA Accelerationcited by 1
  6. 2024KWT-Tiny: RISC-V Accelerated, Embedded Keyword Spotting Transformercited by 0
  7. 2025Adaptive Sparse Softmax: An Effective and Efficient Softmax Variantcited by 0
  8. 2026Techniques for Peak Memory Reduction for LoRA Fine-tuning of LLMs on Edge Devicescited by 0
  9. 2026Recent Developments in Transformer Inference Deployment on FPGA Platforms: A Surveycited by 0