# Uncertainty Estimation for the Open-Set Text Classification systems

arXiv: https://arxiv.org/abs/2604.08560 · HTML: https://mlsanity.com/paper/2604.08560/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 19 of its 38 references that are in the map.
- First posted on arXiv 17 Mar 2026.

## Builds on (most cited first, 19 of 19)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning](https://mlsanity.com/paper/1506.02142/) (2015)
- [Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles](https://mlsanity.com/paper/1612.01474/) (2016)
- [What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?](https://mlsanity.com/paper/1703.04977/) (2017)
- [Towards Open Set Deep Networks](https://mlsanity.com/paper/1511.06233/) (2015)
- [An Evaluation Dataset for Intent Classification and Out-of-Scope Prediction](https://mlsanity.com/paper/1909.02027/) (2019)
- [Recent Advances in Open Set Recognition: A Survey](https://mlsanity.com/paper/1811.08581/) (2018)
- [Probabilistic Face Embeddings](https://mlsanity.com/paper/1904.09658/) (2019)
- [LM-Polygraph: Uncertainty Estimation for Language Models](https://mlsanity.com/paper/2311.07383/) (2023)
- [DOC: Deep Open Classification of Text Documents](https://mlsanity.com/paper/1709.08716/) (2017)
- [Deep Unknown Intent Detection with Margin Loss](https://mlsanity.com/paper/1906.00434/) (2019)
- [Open-world Machine Learning: Applications, Challenges, and Opportunities](https://mlsanity.com/paper/2105.13448/) (2021)
- [Toward Open-Set Face Recognition](https://mlsanity.com/paper/1705.01567/) (2017)
- [VoxBlink2: A 100K+ Speaker Recognition Corpus and the Open-Set Speaker-Identification Benchmark](https://mlsanity.com/paper/2407.11510/) (2024)
- [Hallucination Detection in LLMs with Topological Divergence on Attention Graphs](https://mlsanity.com/paper/2504.10063/) (2025)
- [Stating Comparison Score Uncertainty and Verification Decision Confidence Towards Transparent Face Recognition](https://mlsanity.com/paper/2210.10354/) (2022)
- [Open Set Authorship Attribution toward Demystifying Victorian Periodicals](https://mlsanity.com/paper/1912.08259/) (2019)
- [Holistic Uncertainty Estimation For Open-Set Recognition](https://mlsanity.com/paper/2408.14229/) (2024)

## Abstract (arXiv)

Accurate uncertainty estimation is essential for building robust and trustworthy recognition systems. In this paper, we consider the open-set text classification (OSTC) task - and uncertainty estimation for it. For OSTC a text sample should be classified as one of the existing classes or rejected as unknown. To account for the different uncertainty types encountered in OSTC, we adapt the Holistic Uncertainty Estimation (HolUE) method for the text domain. Our approach addresses two major causes of prediction errors in text recognition systems: text uncertainty that stems from ill formulated queries and gallery uncertainty that is related the ambiguity of data distribution. By capturing these sources, it becomes possible to predict when the system will make a recognition error. We propose a new OSTC benchmark and conduct extensive experiments on a wide range of data, utilizing the authorship attribution, intent and topic classification datasets. HolUE achieves 40-365% improvement in Prediction Rejection Ratio (PRR) over the quality-based SCF baseline across datasets: 365% on Yahoo Answers (0.79 vs 0.17 at FPIR 0.1), 347% on DBPedia (0.85 vs 0.19), 240% on PAN authorship attribution (0.51 vs 0.15 at FPIR 0.5), and 40% on CLINC150 intent classification (0.73 vs~0.52). We make public our code and protocols https://github.com/Leonid-Erlygin/text_uncertainty.git