mlsanityData to 7 Oct 2026

Softermax: Hardware/Software Co-Design of an Efficient Softmax for Transformers

arXiv 2103.09301 ↗First posted 16 Mar 2021cs.ARcited by 52

Not on the map. Why?

Abstract (arXiv)

Transformers have transformed the field of natural language processing. This performance is largely attributed to the use of stacked self-attention layers, each of which consists of matrix multiplies as well as softmax operations. As a result, unlike other neural networks, the softmax operation accounts for a significant fraction of the total run-time of Transformers. To address this, we propose Softermax, a hardware-friendly softmax design. Softermax consists of base replacement, low-precision softmax computations, and an online normalization calculation. We show Softermax results in 2.35x the energy efficiency at 0.90x the size of a comparable baseline, with negligible impact on network accuracy.

Key facts. Cited by 52 papers in the map (data through 7 Oct 2026). First posted on arXiv 16 Mar 2021.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 2; 2022: 6; 2023: 5; 2024: 17; 2025: 13; 2026: 9.

52 papers.

Sort:

By area:

  1. 2022I-ViT: Integer-only Quantization for Efficient Vision Transformer Inferencecited by 93
  2. 2023Full Stack Optimization of Transformer Inference: a Surveycited by 79
  3. 2023Generative Pre-trained Transformer: A Comprehensive Review on Enabling Technologies, Potential Applications, Emerging Challenges, and Future Directionscited by 55
  4. 2022SimA: Simple Softmax-free Attention for Vision Transformerscited by 32
  5. 2021NN-LUT: Neural Approximation of Non-Linear Operations for Efficient Transformer Inferencecited by 25
  6. 2023X-Former: In-Memory Acceleration of Transformerscited by 25
  7. 2025Vision Transformers on the Edge: A Comprehensive Survey of Model Compression and Acceleration Strategiescited by 24
  8. 2024I-LLM: Efficient Integer-Only Inference for Fully-Quantized Low-Bit Large Language Modelscited by 19
  9. 2023Softmax Bias Correction for Quantized Generative Modelscited by 15
  10. 2024Flex-PE: Flexible and SIMD Multi-Precision Processing Element for AI Workloadscited by 15
  11. 2024ConSmax: Hardware-Friendly Alternative Softmax with Learnable Parameterscited by 14
  12. 2024Model Quantization and Hardware Acceleration for Vision Transformers: A Comprehensive Surveycited by 13
38 more
  1. 2024PEANO-ViT: Power-Efficient Approximations of Non-Linearities in Vision Transformerscited by 13
  2. 2022Sparse Attention Acceleration with Synergistic In-Memory Pruning and On-Chip Recomputationcited by 12
  3. 2023ITA: An Energy-Efficient Attention and Softmax Accelerator for Quantized Transformerscited by 12
  4. 2022SALO: An Efficient Spatial Accelerator Enabling Hybrid Sparse Attention Mechanisms for Long Sequencescited by 10
  5. 2025QUARK: Quantization-Enabled Circuit Sharing for Transformer Acceleration by Exploiting Common Patterns in Nonlinear Operationscited by 10
  6. 2021Efficient Softmax Approximation for Deep Neural Networks with Attention Mechanismcited by 9
  7. 2022Accelerating Attention through Gradient-Based Learned Runtime Pruningcited by 8
  8. 2025LUT-DLA: Lookup Table as Efficient Extreme Low-Bit Deep Learning Acceleratorcited by 8
  9. 2024Integer-only Quantized Transformers for Embedded FPGA-based Time-series Forecasting in AIoTcited by 6
  10. 2025Low-Cost FlashAttention with Fused Exponential and Multiplication Hardware Operatorscited by 5
  11. 2025Kelle: Co-design KV Caching and eDRAM for Efficient LLM Serving in Edge Computingcited by 5
  12. 2025SOLE: Hardware-Software Co-design of Softmax and LayerNorm for Efficient Transformer Inferencecited by 5
  13. 2025VEXP: A Low-Cost RISC-V ISA Extension for Accelerated Softmax Computation in Transformerscited by 4
  14. 2022TrimBERT: Tailoring BERT for Trade-offscited by 3
  15. 2024Reusing Softmax Hardware Unit for GELU Computation in Transformerscited by 3
  16. 2024OPAL: Outlier-Preserved Microscaling Quantization Accelerator for Generative Large Language Modelscited by 3
  17. 2024EXAQ: Exponent Aware Quantization For LLMs Accelerationcited by 3
  18. 2024SoftmAP: Software-Hardware Co-design for Integer-Only Softmax on Associative Processorscited by 3
  19. 2025IntAttention: A Fully Integer Attention Pipeline for Efficient Edge Inferencecited by 3
  20. 2024Genetic Quantization-Aware Approximation for Non-Linear Operations in Transformerscited by 2
  21. 2024Understanding the Performance Horizon of the Latest ML Workloads with NonGEMM Workloadscited by 2
  22. 2024From Algorithm to Hardware: A Survey on Efficient and Safe Deployment of Deep Neural Networkscited by 2
  23. 2024TReX- Reusing Vision Transformer's Attention for Efficient Xbar-based Computingcited by 2
  24. 2025VEDA: Efficient LLM Generation Through Voting-based KV Cache Eviction and Dataflow-flexible Acceleratorcited by 1
  25. 2025SpeLLM: Character-Level Multi-Head Decodingcited by 1
  26. 2026BAPS: A Fine-Grained Low-Precision Scheme for Softmax in Attention via Block-Aware Precision reScalingcited by 1
  27. 2026NLI:Non-uniform Linear Interpolation Approximation of Nonlinear Operations for Efficient LLMs Inferencecited by 1
  28. 2024Faster Inference of Integer SWIN Transformer by Removing the GELU Activationcited by 0
  29. 2024NOVA: NoC-based Vector Unit for Mapping Attention Layers on a CNN Acceleratorcited by 0
  30. 2024QuAKE: Speeding up Model Inference Using Quick and Approximate Kernels for Exponential Non-Linearitiescited by 0
  31. 2025Efficient FPGA Implementation of Time-Domain Popcount for Low-Complexity Machine Learningcited by 0
  32. 2025FAR: Function-preserving Attention Replacement for IMC-friendly Inferencecited by 0
  33. 2025Vectorized FlashAttention with Low-cost Exponential Computation in RISC-V Vector Processorscited by 0
  34. 2026Mugi: Value Level Parallelism For Efficient LLMscited by 0
  35. 2026Taming the Exponential: A Fast Softmax Surrogate for Integer-Native Edge Inferencecited by 0
  36. 2026Hardware-Efficient Softmax and Layer Normalization with Guaranteed Normalization for Edge Devicescited by 0
  37. 2026MIVE: A Minimalist Integer Vector Engine for Softmax LayerNorm and RMSNorm Accelerationcited by 0
  38. 2026Compute-in-Memory Attention: A Time-Domain Analog Softmax Circuit with RC-Tunable Temperaturecited by 0

50 most cited of 52 All 52 (JSON)