# Taming the Exponential: A Fast Softmax Surrogate for Integer-Native Edge Inference

arXiv: https://arxiv.org/abs/2604.02292 · HTML: https://mlsanity.com/paper/2604.02292/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 15 of its 26 references that are in the map.
- First posted on arXiv 2 Apr 2026.

## Builds on (most cited first, 15 of 15)

- [A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference](https://mlsanity.com/paper/1704.05426/) (2017)
- [From Softmax to Sparsemax: A Sparse Model of Attention and Multi-Label Classification](https://mlsanity.com/paper/1602.02068/) (2016)
- [I-BERT: Integer-only BERT Quantization](https://mlsanity.com/paper/2101.01321/) (2021)
- [Sparse Sequence-to-Sequence Models](https://mlsanity.com/paper/1905.05702/) (2019)
- [Softermax: Hardware/Software Co-Design of an Efficient Softmax for Transformers](https://mlsanity.com/paper/2103.09301/) (2021)
- [Sparse Attention with Linear Units](https://mlsanity.com/paper/2104.07012/) (2021)
- [ConSmax: Hardware-Friendly Alternative Softmax with Learnable Parameters](https://mlsanity.com/paper/2402.10930/) (2024)
- [ITA: An Energy-Efficient Attention and Softmax Accelerator for Quantized Transformers](https://mlsanity.com/paper/2307.03493/) (2023)
- [Efficiency, Expressivity, and Extensibility in a Close-to-Metal NPU Programming Interface](https://mlsanity.com/paper/2504.18430/) (2025)
- [hls4ml: A Flexible, Open-Source Platform for Deep Learning Acceleration on Reconfigurable Hardware](https://mlsanity.com/paper/2512.01463/) (2025)
- [TransAxx: Efficient Transformers with Approximate Computing](https://mlsanity.com/paper/2402.07545/) (2024)
- [Low Latency Transformer Inference on FPGAs for Physics Applications with hls4ml](https://mlsanity.com/paper/2409.05207/) (2024)
- [TurboAttention: Efficient Attention Approximation For High Throughputs LLMs](https://mlsanity.com/paper/2412.08585/) (2024)
- [AIE4ML: An End-to-End Framework for Compiling Neural Networks for the Next Generation of AMD AI Engines](https://mlsanity.com/paper/2512.15946/) (2025)
- [IntAttention: A Fully Integer Attention Pipeline for Efficient Edge Inference](https://mlsanity.com/paper/2511.21513/) (2025)

## Abstract (arXiv)

Softmax can become a computational bottleneck in the Transformer model's Multi-Head Attention (MHA) block, particularly in small models under low-precision inference, where exponentiation and normalization incur significant overhead. As such, we suggest using Head-Calibrated Clipped-Linear Softmax (HCCS), a bounded, monotone surrogate to the exponential softmax function, which uses a clipped linear mapping of the max centered attention logits. This approximation produces a stable probability distribution, maintains the ordering of the original logits and has non-negative values. HCCS differs from previous softmax surrogates as it includes a set of lightweight calibration parameters that are optimized offline based on a representative dataset and calibrated for each individual attention head to preserve the statistical properties of the individual heads. We describe a hardware-motivated implementation of HCCS for high-throughput scenarios targeting the AMD Versal AI Engines. The current reference implementations from AMD for this platform rely upon either bfloat16 arithmetic or LUTs to perform the exponential operation, which might limit the throughput of the platform and fail to utilize the high-throughput integer vector processing units of the AI Engine. In contrast, HCCS provides a natural mapping to the AI Engines' int8 multiply accumulate (MAC) units. To the best of our knowledge, this is the first int8 optimized softmax surrogate for AMD AI engines that significantly exceeds the speed performance of other reference implementations while maintaining competitive task accuracy on small or heavily quantized MHA workloads after quantization-aware retraining.