# On Controllable Sparse Alternatives to Softmax

arXiv: https://arxiv.org/abs/1810.11975 · HTML: https://mlsanity.com/paper/1810.11975/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 Oct 2018.

## Built on by (most cited first, 27 of 27)

- [Is Sparse Attention more Interpretable?](https://mlsanity.com/paper/2106.01087/) (2021)
- [Softmax is 1/2-Lipschitz: A tight bound across all ℓp norms](https://mlsanity.com/paper/2510.23012/) (2025)
- [LD-MoLE: Learnable Dynamic Routing for Mixture of LoRA Experts](https://mlsanity.com/paper/2509.25684/) (2025)
- [Pay Attention to Attention Distribution: A New Local Lipschitz Bound for Transformers](https://mlsanity.com/paper/2507.07814/) (2025)
- [The Lipschitz-Variance-Margin Tradeoff for Enhanced Randomized Smoothing](https://mlsanity.com/paper/2309.16883/) (2023)
- [Evidential Softmax for Sparse Multimodal Distributions in Deep Generative Models](https://mlsanity.com/paper/2110.14182/) (2021)
- [On the Interpretability of Attention Networks](https://mlsanity.com/paper/2212.14776/) (2022)
- [Understanding Gaussian Attention Bias of Vision Transformers Using Effective Receptive Fields](https://mlsanity.com/paper/2305.04722/) (2023)
- [MultiMax: Sparse and Multi-Modal Attention Learning](https://mlsanity.com/paper/2406.01189/) (2024)
- [Discourse-Aware Soft Prompting for Text Generation](https://mlsanity.com/paper/2112.05717/) (2021)
- [Yet Another Representation of Binary Decision Trees: A Mathematical Demonstration](https://mlsanity.com/paper/2101.07077/) (2021)
- [Sparse Communication via Mixed Distributions](https://mlsanity.com/paper/2108.02658/) (2021)
- [Beyond the Policy Gradient Theorem for Efficient Policy Updates in Actor-Critic Algorithms](https://mlsanity.com/paper/2202.07496/) (2022)
- [Exploiting Field Dependencies for Learning on Categorical Data](https://mlsanity.com/paper/2307.09321/) (2023)
- [Distance-Ratio-Based Formulation for Metric Learning](https://mlsanity.com/paper/2201.08676/) (2022)
- [Filling out the missing gaps: Time Series Imputation with Semi-Supervised Learning](https://mlsanity.com/paper/2304.04275/) (2023)
- [r-softmax: Generalized Softmax with Controllable Sparsity Rate](https://mlsanity.com/paper/2304.05243/) (2023)
- [Near-Linear Time Projection onto the ℓ1,∞ Ball; Application to Sparse Autoencoders](https://mlsanity.com/paper/2307.09836/) (2023)
- [MixLight: Borrowing the Best of both Spherical Harmonics and Gaussian Models](https://mlsanity.com/paper/2404.12768/) (2024)
- [Probability Bounding: Post-Hoc Calibration via Box-Constrained Softmax](https://mlsanity.com/paper/2506.10572/) (2025)
- [When Should Graph Attention Be Sparse? Learning a Per-Edge Tsallis Index](https://mlsanity.com/paper/2608.02938/) (2026)
- [Decision Machines: Congruent Decision Trees](https://mlsanity.com/paper/2101.11347/) (2021)
- [SPMoE: Generate Multiple Pattern-Aware Outputs with Sparse Pattern Mixture of Experts](https://mlsanity.com/paper/2108.07535/) (2021)
- [Explainability of Text Processing and Retrieval Methods: A Survey](https://mlsanity.com/paper/2212.07126/) (2022)
- [Accurate and Efficient Multi-Channel Time Series Forecasting via Sparse Attention Mechanism](https://mlsanity.com/paper/2603.18712/) (2026)
- [ExpertHTR: Unified Handwritten Text Recognition with Multi-Task Learning and Sparse Mixture-of-Experts](https://mlsanity.com/paper/2609.12705/) (2026)
- [Sparse All-Layer Connector for Domain Generalised Semantic Segmentation](https://mlsanity.com/paper/2610.15660/) (2026)

## Abstract (arXiv)

Converting an n-dimensional vector to a probability distribution over n objects is a commonly used component in many machine learning tasks like multiclass classification, multilabel classification, attention mechanisms etc. For this, several probability mapping functions have been proposed and employed in literature such as softmax, sum-normalization, spherical softmax, and sparsemax, but there is very little understanding in terms how they relate with each other. Further, none of the above formulations offer an explicit control over the degree of sparsity. To address this, we develop a unified framework that encompasses all these formulations as special cases. This framework ensures simple closed-form solutions and existence of sub-gradients suitable for learning via backpropagation. Within this framework, we propose two novel sparse formulations, sparsegen-lin and sparsehourglass, that seek to provide a control over the degree of desired sparsity. We further develop novel convex loss functions that help induce the behavior of aforementioned formulations in the multilabel classification setting, showing improved performance. We also demonstrate empirically that the proposed formulations, when used to compute attention weights, achieve better or comparable performance on standard seq2seq tasks like neural machine translation and abstractive summarization.