# Revisiting Softmax for Uncertainty Approximation in Text Classification

arXiv: https://arxiv.org/abs/2210.14037 · HTML: https://mlsanity.com/paper/2210.14037/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 40 references that are in the map.
- First posted on arXiv 25 Oct 2022.

## Builds on (most cited first, 19 of 19)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning](https://mlsanity.com/paper/1506.02142/) (2015)
- [Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles](https://mlsanity.com/paper/1612.01474/) (2016)
- [A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks](https://mlsanity.com/paper/1610.02136/) (2016)
- [Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift](https://mlsanity.com/paper/1906.02530/) (2019)
- [Energy and Policy Considerations for Deep Learning in NLP](https://mlsanity.com/paper/1906.02243/) (2019)
- [A Survey of Uncertainty in Deep Neural Networks](https://mlsanity.com/paper/2107.03342/) (2021)
- [Carbon Emissions and Large Neural Network Training](https://mlsanity.com/paper/2104.10350/) (2021)
- [A Survey of Transformers](https://mlsanity.com/paper/2106.04554/) (2021)
- [Masksembles for Uncertainty Estimation](https://mlsanity.com/paper/2012.08334/) (2020)
- [Being Bayesian about Categorical Probability](https://mlsanity.com/paper/2002.07965/) (2020)
- [Are Pre-trained Convolutions Better than Pre-trained Transformers?](https://mlsanity.com/paper/2105.03322/) (2021)
- [Mitigating Uncertainty in Document Classification](https://mlsanity.com/paper/1907.07590/) (2019)
- [Inhibited Softmax for Uncertainty Estimation in Neural Networks](https://mlsanity.com/paper/1810.01861/) (2018)
- [Citation Needed: A Taxonomy and Algorithmic Assessment of Wikipedia's Verifiability](https://mlsanity.com/paper/1902.11116/) (2019)
- [Dropout as a Bayesian Approximation: Appendix](https://mlsanity.com/paper/1506.02157/) (2015)

## Built on by (most cited first, 4 of 4)

- [Non-Exchangeable Conformal Language Generation with Nearest Neighbors](https://mlsanity.com/paper/2402.00707/) (2024)
- [Revisiting Softmax Masking: Stop Gradient for Enhancing Stability in Replay-based Continual Learning](https://mlsanity.com/paper/2309.14808/) (2023)
- [Noise reduction in BERT NER models for clinical entity extraction](https://mlsanity.com/paper/2603.00022/) (2026)
- [Plug-in Losses for Evidential Deep Learning: A Simplified Framework for Uncertainty Estimation that Includes the Softmax Classifier](https://mlsanity.com/paper/2605.22746/) (2026)

## Abstract (arXiv)

Uncertainty approximation in text classification is an important area with applications in domain adaptation and interpretability. One of the most widely used uncertainty approximation methods is Monte Carlo (MC) Dropout, which is computationally expensive as it requires multiple forward passes through the model. A cheaper alternative is to simply use the softmax based on a single forward pass without dropout to estimate model uncertainty. However, prior work has indicated that these predictions tend to be overconfident. In this paper, we perform a thorough empirical analysis of these methods on five datasets with two base neural architectures in order to identify the trade-offs between the two. We compare both softmax and an efficient version of MC Dropout on their uncertainty approximations and downstream text classification performance, while weighing their runtime (cost) against performance (benefit). We find that, while MC dropout produces the best uncertainty approximations, using a simple softmax leads to competitive and in some cases better uncertainty estimation for text classification at a much lower computational cost, suggesting that softmax can in fact be a sufficient uncertainty estimate when computational resources are a concern.