# Estimating Confidence of Predictions of Individual Classifiers and Their Ensembles for the Genre Classification Task

arXiv: https://arxiv.org/abs/2206.07427 · HTML: https://mlsanity.com/paper/2206.07427/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 7 of its 16 references that are in the map.
- First posted on arXiv 15 Jun 2022.

## Builds on (most cited first, 7 of 7)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [How to Fine-Tune BERT for Text Classification?](https://mlsanity.com/paper/1905.05583/) (2019)
- [Adaptation of Deep Bidirectional Multilingual Transformers for Russian Language](https://mlsanity.com/paper/1905.07213/) (2019)
- [Dropout Strikes Back: Improved Uncertainty Estimation via Diversity Sampling](https://mlsanity.com/paper/2003.03274/) (2020)
- [Experiments with adversarial attacks on text genres](https://mlsanity.com/paper/2107.02246/) (2021)

## Built on by (most cited first, 3 of 3)

- [Automatic register identification for the open web using multilingual deep learning](https://mlsanity.com/paper/2406.19892/) (2024)
- [Controlling Out-of-Domain Gaps in LLMs for Genre Classification and Generated Text Detection](https://mlsanity.com/paper/2412.20595/) (2024)
- [Measuring Online Hate on 4chan using Pre-trained Deep Learning Models](https://mlsanity.com/paper/2504.00045/) (2025)

## Abstract (arXiv)

Genre identification is a subclass of non-topical text classification. The main difference between this task and topical classification is that genres, unlike topics, usually do not correspond to simple keywords, and thus they need to be defined in terms of their functions in communication. Neural models based on pre-trained transformers, such as BERT or XLM-RoBERTa, demonstrate SOTA results in many NLP tasks, including non-topical classification. However, in many cases, their downstream application to very large corpora, such as those extracted from social media, can lead to unreliable results because of dataset shifts, when some raw texts do not match the profile of the training set. To mitigate this problem, we experiment with individual models as well as with their ensembles. To evaluate the robustness of all models we use a prediction confidence metric, which estimates the reliability of a prediction in the absence of a gold standard label. We can evaluate robustness via the confidence gap between the correctly classified texts and the misclassified ones on a labeled test corpus, higher gaps make it easier to improve our confidence that our classifier made the right decision. Our results show that for all of the classifiers tested in this study, there is a confidence gap, but for the ensembles, the gap is bigger, meaning that ensembles are more robust than their individual models.