# Softermax: Hardware/Software Co-Design of an Efficient Softmax for Transformers

arXiv: https://arxiv.org/abs/2103.09301 · HTML: https://mlsanity.com/paper/2103.09301/

## Key facts

- Cited by 54 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 Mar 2021.

## Built on by (most cited first, 30 of 54)

- [I-ViT: Integer-only Quantization for Efficient Vision Transformer Inference](https://mlsanity.com/paper/2207.01405/) (2022)
- [Full Stack Optimization of Transformer Inference: a Survey](https://mlsanity.com/paper/2302.14017/) (2023)
- [Generative Pre-trained Transformer: A Comprehensive Review on Enabling Technologies, Potential Applications, Emerging Challenges, and Future Directions](https://mlsanity.com/paper/2305.10435/) (2023)
- [SimA: Simple Softmax-free Attention for Vision Transformers](https://mlsanity.com/paper/2206.08898/) (2022)
- [NN-LUT: Neural Approximation of Non-Linear Operations for Efficient Transformer Inference](https://mlsanity.com/paper/2112.02191/) (2021)
- [X-Former: In-Memory Acceleration of Transformers](https://mlsanity.com/paper/2303.07470/) (2023)
- [Vision Transformers on the Edge: A Comprehensive Survey of Model Compression and Acceleration Strategies](https://mlsanity.com/paper/2503.02891/) (2025)
- [I-LLM: Efficient Integer-Only Inference for Fully-Quantized Low-Bit Large Language Models](https://mlsanity.com/paper/2405.17849/) (2024)
- [Softmax Bias Correction for Quantized Generative Models](https://mlsanity.com/paper/2309.01729/) (2023)
- [Flex-PE: Flexible and SIMD Multi-Precision Processing Element for AI Workloads](https://mlsanity.com/paper/2412.11702/) (2024)
- [ConSmax: Hardware-Friendly Alternative Softmax with Learnable Parameters](https://mlsanity.com/paper/2402.10930/) (2024)
- [Model Quantization and Hardware Acceleration for Vision Transformers: A Comprehensive Survey](https://mlsanity.com/paper/2405.00314/) (2024)
- [PEANO-ViT: Power-Efficient Approximations of Non-Linearities in Vision Transformers](https://mlsanity.com/paper/2406.14854/) (2024)
- [Sparse Attention Acceleration with Synergistic In-Memory Pruning and On-Chip Recomputation](https://mlsanity.com/paper/2209.00606/) (2022)
- [ITA: An Energy-Efficient Attention and Softmax Accelerator for Quantized Transformers](https://mlsanity.com/paper/2307.03493/) (2023)
- [QUARK: Quantization-Enabled Circuit Sharing for Transformer Acceleration by Exploiting Common Patterns in Nonlinear Operations](https://mlsanity.com/paper/2511.06767/) (2025)
- [SALO: An Efficient Spatial Accelerator Enabling Hybrid Sparse Attention Mechanisms for Long Sequences](https://mlsanity.com/paper/2206.14550/) (2022)
- [Efficient Softmax Approximation for Deep Neural Networks with Attention Mechanism](https://mlsanity.com/paper/2111.10770/) (2021)
- [LUT-DLA: Lookup Table as Efficient Extreme Low-Bit Deep Learning Accelerator](https://mlsanity.com/paper/2501.10658/) (2025)
- [Accelerating Attention through Gradient-Based Learned Runtime Pruning](https://mlsanity.com/paper/2204.03227/) (2022)
- [Integer-only Quantized Transformers for Embedded FPGA-based Time-series Forecasting in AIoT](https://mlsanity.com/paper/2407.11041/) (2024)
- [Low-Cost FlashAttention with Fused Exponential and Multiplication Hardware Operators](https://mlsanity.com/paper/2505.14314/) (2025)
- [Kelle: Co-design KV Caching and eDRAM for Efficient LLM Serving in Edge Computing](https://mlsanity.com/paper/2510.16040/) (2025)
- [SOLE: Hardware-Software Co-design of Softmax and LayerNorm for Efficient Transformer Inference](https://mlsanity.com/paper/2510.17189/) (2025)
- [MEGA: A Memory-Efficient GNN Accelerator Exploiting Degree-Aware Mixed-Precision Quantization](https://mlsanity.com/paper/2311.09775/) (2023)
- [Understanding the Performance Horizon of the Latest ML Workloads with NonGEMM Workloads](https://mlsanity.com/paper/2404.11788/) (2024)
- [OPAL: Outlier-Preserved Microscaling Quantization Accelerator for Generative Large Language Models](https://mlsanity.com/paper/2409.05902/) (2024)
- [SoftmAP: Software-Hardware Co-design for Integer-Only Softmax on Associative Processors](https://mlsanity.com/paper/2411.17847/) (2024)
- [VEXP: A Low-Cost RISC-V ISA Extension for Accelerated Softmax Computation in Transformers](https://mlsanity.com/paper/2504.11227/) (2025)
- [TrimBERT: Tailoring BERT for Trade-offs](https://mlsanity.com/paper/2202.12411/) (2022)

## Abstract (arXiv)

Transformers have transformed the field of natural language processing. This performance is largely attributed to the use of stacked self-attention layers, each of which consists of matrix multiplies as well as softmax operations. As a result, unlike other neural networks, the softmax operation accounts for a significant fraction of the total run-time of Transformers. To address this, we propose Softermax, a hardware-friendly softmax design. Softermax consists of base replacement, low-precision softmax computations, and an online normalization calculation. We show Softermax results in 2.35x the energy efficiency at 0.90x the size of a comparable baseline, with negligible impact on network accuracy.