# An Exploration of Softmax Alternatives Belonging to the Spherical Loss Family

arXiv: https://arxiv.org/abs/1511.05042 · HTML: https://mlsanity.com/paper/1511.05042/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 Nov 2015.

## Built on by (most cited first, 27 of 27)

- [Low-Cost High-Power Membership Inference Attacks](https://mlsanity.com/paper/2312.03262/) (2023)
- [von Mises-Fisher Loss: An Exploration of Embedding Geometries for Supervised Learning](https://mlsanity.com/paper/2103.15718/) (2021)
- [SMART: Submodular Data Mixture Strategy for Instruction Tuning](https://mlsanity.com/paper/2403.08370/) (2024)
- [ConSmax: Hardware-Friendly Alternative Softmax with Learnable Parameters](https://mlsanity.com/paper/2402.10930/) (2024)
- [Universal Spam Detection using Transfer Learning of BERT Model](https://mlsanity.com/paper/2202.03480/) (2022)
- [On the Lipschitz Constant of Deep Networks and Double Descent](https://mlsanity.com/paper/2301.12309/) (2023)
- [MILO: Model-Agnostic Subset Selection Framework for Efficient Model Training and Tuning](https://mlsanity.com/paper/2301.13287/) (2023)
- [Fast Variational AutoEncoder with Inverted Multi-Index for Collaborative Filtering](https://mlsanity.com/paper/2109.05773/) (2021)
- [Building a stable classifier with the inflated argmax](https://mlsanity.com/paper/2405.14064/) (2024)
- [MANO: Exploiting Matrix Norm for Unsupervised Accuracy Estimation Under Distribution Shifts](https://mlsanity.com/paper/2405.18979/) (2024)
- [Escaping the Gradient Vanishing: Periodic Alternatives of Softmax in Attention Mechanism](https://mlsanity.com/paper/2108.07153/) (2021)
- [TaylorShift: Shifting the Complexity of Self-Attention from Squared to Linear (and Back) using Taylor-Softmax](https://mlsanity.com/paper/2403.02920/) (2024)
- [On the Interpretability of Attention Networks](https://mlsanity.com/paper/2212.14776/) (2022)
- [Instance-Level Data-Use Auditing of Visual ML Models](https://mlsanity.com/paper/2503.22413/) (2025)
- [HyTRec: A Hybrid Temporal-Aware Attention Architecture for Long Behavior Sequential Recommendation](https://mlsanity.com/paper/2602.18283/) (2026)
- [Generalizing Speaker Verification for Spoof Awareness in the Embedding Space](https://mlsanity.com/paper/2401.11156/) (2024)
- [Sparse-softmax: A Simpler and Faster Alternative Softmax Transformation](https://mlsanity.com/paper/2112.12433/) (2021)
- [A Large Deviations Perspective on Policy Gradient Algorithms](https://mlsanity.com/paper/2311.07411/) (2023)
- [Adaptive Temperature Based on Logits Correlation in Knowledge Distillation](https://mlsanity.com/paper/2503.09030/) (2025)
- [Enhancing Classifier Conservativeness and Robustness by Polynomiality](https://mlsanity.com/paper/2203.12693/) (2022)
- [r-softmax: Generalized Softmax with Controllable Sparsity Rate](https://mlsanity.com/paper/2304.05243/) (2023)
- [Flexformer: Flexible Linear Transformer with Learnable Attention Kernel](https://mlsanity.com/paper/2606.27748/) (2026)
- [A novel multi-scale loss function for classification problems in machine learning](https://mlsanity.com/paper/2106.02676/) (2021)
- [Unfolding Projection-free SDP Relaxation of Binary Graph Classifier via GDPA Linearization](https://mlsanity.com/paper/2109.04697/) (2021)
- [Classified as unknown: A novel Bayesian neural network](https://mlsanity.com/paper/2301.13401/) (2023)
- [An extended asymmetric sigmoid with Perceptron (SIGTRON) for imbalanced linear classification](https://mlsanity.com/paper/2312.16043/) (2023)
- [Adaptive Sparse Softmax: An Effective and Efficient Softmax Variant](https://mlsanity.com/paper/2508.03175/) (2025)

## Abstract (arXiv)

In a multi-class classification problem, it is standard to model the output of a neural network as a categorical distribution conditioned on the inputs. The output must therefore be positive and sum to one, which is traditionally enforced by a softmax. This probabilistic mapping allows to use the maximum likelihood principle, which leads to the well-known log-softmax loss. However the choice of the softmax function seems somehow arbitrary as there are many other possible normalizing functions. It is thus unclear why the log-softmax loss would perform better than other loss alternatives. In particular Vincent et al. (2015) recently introduced a class of loss functions, called the spherical family, for which there exists an efficient algorithm to compute the updates of the output weights irrespective of the output size. In this paper, we explore several loss functions from this family as possible alternatives to the traditional log-softmax. In particular, we focus our investigation on spherical bounds of the log-softmax loss and on two spherical log-likelihood losses, namely the log-Spherical Softmax suggested by Vincent et al. (2015) and the log-Taylor Softmax that we introduce. Although these alternatives do not yield as good results as the log-softmax loss on two language modeling tasks, they surprisingly outperform it in our experiments on MNIST and CIFAR-10, suggesting that they might be relevant in a broad range of applications.