# Exploring Alternatives to Softmax Function

arXiv: https://arxiv.org/abs/2011.11538 · HTML: https://mlsanity.com/paper/2011.11538/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 Nov 2020.

## Built on by (most cited first, 16 of 16)

- [The Hedgehog & the Porcupine: Expressive Linear Attentions with Softmax Mimicry](https://mlsanity.com/paper/2402.04347/) (2024)
- [Low-Cost High-Power Membership Inference Attacks](https://mlsanity.com/paper/2312.03262/) (2023)
- [SimA: Simple Softmax-free Attention for Vision Transformers](https://mlsanity.com/paper/2206.08898/) (2022)
- [The Fine-Grained Complexity of Gradient Computation for Training Large Language Models](https://mlsanity.com/paper/2402.04497/) (2024)
- [ConSmax: Hardware-Friendly Alternative Softmax with Learnable Parameters](https://mlsanity.com/paper/2402.10930/) (2024)
- [Fast RoPE Attention: Combining the Polynomial Method and Fast Fourier Transform](https://mlsanity.com/paper/2505.11892/) (2025)
- [Rethinking Attention: Polynomial Alternatives to Softmax in Transformers](https://mlsanity.com/paper/2410.18613/) (2024)
- [Escaping the Gradient Vanishing: Periodic Alternatives of Softmax in Attention Mechanism](https://mlsanity.com/paper/2108.07153/) (2021)
- [TaylorShift: Shifting the Complexity of Self-Attention from Squared to Linear (and Back) using Taylor-Softmax](https://mlsanity.com/paper/2403.02920/) (2024)
- [Parallel Sequence Modeling via Generalized Spatial Propagation Network](https://mlsanity.com/paper/2501.12381/) (2025)
- [A Neural Network Based Choice Model for Assortment Optimization](https://mlsanity.com/paper/2308.05617/) (2023)
- [Adaptive Temperature Based on Logits Correlation in Knowledge Distillation](https://mlsanity.com/paper/2503.09030/) (2025)
- [Untangling Component Imbalance in Hybrid Linear Attention Conversion Methods](https://mlsanity.com/paper/2510.05901/) (2025)
- [Unleashing the power of Neural Collapse for Transferability Estimation](https://mlsanity.com/paper/2310.05754/) (2023)
- [Classified as unknown: A novel Bayesian neural network](https://mlsanity.com/paper/2301.13401/) (2023)
- [Is Softmax Loss All You Need? A Principled Analysis of Softmax-family Loss](https://mlsanity.com/paper/2601.22745/) (2026)

## Abstract (arXiv)

Softmax function is widely used in artificial neural networks for multiclass classification, multilabel classification, attention mechanisms, etc. However, its efficacy is often questioned in literature. The log-softmax loss has been shown to belong to a more generic class of loss functions, called spherical family, and its member log-Taylor softmax loss is arguably the best alternative in this class. In another approach which tries to enhance the discriminative nature of the softmax function, soft-margin softmax (SM-softmax) has been proposed to be the most suitable alternative. In this work, we investigate Taylor softmax, SM-softmax and our proposed SM-Taylor softmax, an amalgamation of the earlier two functions, as alternatives to softmax function. Furthermore, we explore the effect of expanding Taylor softmax up to ten terms (original work proposed expanding only to two terms) along with the ramifications of considering Taylor softmax to be a finite or infinite series during backpropagation. Our experiments for the image classification task on different datasets reveal that there is always a configuration of the SM-Taylor softmax function that outperforms the normal softmax function and its other alternatives.