# Are you sure? Measuring models bias in content moderation through uncertainty

arXiv: https://arxiv.org/abs/2509.22699 · HTML: https://mlsanity.com/paper/2509.22699/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 62 references that are in the map.
- First posted on arXiv 21 Sep 2025.

## Builds on (most cited first, 30 of 30)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [BLOOM: A 176B-Parameter Open-Access Multilingual Language Model](https://mlsanity.com/paper/2211.05100/) (2022)
- [Semantics derived automatically from language corpora contain human-like biases](https://mlsanity.com/paper/1608.07187/) (2016)
- [A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantification](https://mlsanity.com/paper/2107.07511/) (2021)
- [Large Language Models](https://mlsanity.com/paper/2307.05782/) (2023)
- [Whose Opinions Do Language Models Reflect?](https://mlsanity.com/paper/2303.17548/) (2023)
- [OLMo: Accelerating the Science of Language Models](https://mlsanity.com/paper/2402.00838/) (2024)
- [Social Bias Frames: Reasoning about Social and Power Implications of Language](https://mlsanity.com/paper/1911.03891/) (2019)
- [Annotators with Attitudes: How Annotator Beliefs And Identities Bias Toxic Language Detection](https://mlsanity.com/paper/2111.07997/) (2021)
- [A Survey of Confidence Estimation and Calibration in Large Language Models](https://mlsanity.com/paper/2311.08298/) (2023)
- [Learning from the Worst: Dynamically Generated Datasets to Improve Online Hate Detection](https://mlsanity.com/paper/2012.15761/) (2020)
- [Evaluating the Moral Beliefs Encoded in LLMs](https://mlsanity.com/paper/2307.14324/) (2023)
- [HateBERT: Retraining BERT for Abusive Language Detection in English](https://mlsanity.com/paper/2010.12472/) (2020)
- [Assessing Cross-Cultural Alignment between ChatGPT and Human Societies: An Empirical Study](https://mlsanity.com/paper/2303.17466/) (2023)
- [RedditBias: A Real-World Resource for Bias Evaluation and Debiasing of Conversational Language Models](https://mlsanity.com/paper/2106.03521/) (2021)
- [Deceiving Google's Perspective API Built for Detecting Toxic Comments](https://mlsanity.com/paper/1702.08138/) (2017)
- [Evaluating the Underlying Gender Bias in Contextualized Word Embeddings](https://mlsanity.com/paper/1904.08783/) (2019)
- [When the Majority is Wrong: Modeling Annotator Disagreement for Subjective Tasks](https://mlsanity.com/paper/2305.06626/) (2023)
- [Quantifying Uncertainties in Natural Language Processing Tasks](https://mlsanity.com/paper/1811.07253/) (2018)
- [Understanding Undesirable Word Embedding Associations](https://mlsanity.com/paper/1908.06361/) (2019)
- [Conformal Prediction for Natural Language Processing: A Survey](https://mlsanity.com/paper/2405.01976/) (2024)
- [Exploring Cross-Cultural Differences in English Hate Speech Annotations: From Dataset Construction to Analysis](https://mlsanity.com/paper/2308.16705/) (2023)
- [Measuring Representational Harms in Image Captioning](https://mlsanity.com/paper/2206.07173/) (2022)
- [Assessing LLMs for Moral Value Pluralism](https://mlsanity.com/paper/2312.10075/) (2023)
- [Robust Hate Speech Detection in Social Media: A Cross-Dataset Empirical Evaluation](https://mlsanity.com/paper/2307.01680/) (2023)
- [Data Bootstrapping Approaches to Improve Low Resource Abusive Language Detection for Indic Languages](https://mlsanity.com/paper/2204.12543/) (2022)
- [Don't Blame the Data, Blame the Model: Understanding Noise and Bias When Learning from Subjective Annotations](https://mlsanity.com/paper/2403.04085/) (2024)
- [pysentimiento: A Python Toolkit for Opinion Mining and Social NLP tasks](https://mlsanity.com/paper/2106.09462/) (2021)
- [Logit-based Uncertainty Measure in Classification](https://mlsanity.com/paper/2107.02845/) (2021)
- [A Collaborative Content Moderation Framework for Toxicity Detection based on Conformalized Estimates of Annotation Disagreement](https://mlsanity.com/paper/2411.04090/) (2024)

## Built on by (most cited first, 2 of 2)

- [U-SEG: Uncertainty in SEGmentation -- A systematic multi-variable exploration](https://mlsanity.com/paper/2605.15421/) (2026)
- [The Ghost Annotator: a Framework to Explore Human Label Variation in Content Moderation through Conformal Prediction](https://mlsanity.com/paper/2606.02911/) (2026)

## Abstract (arXiv)

Automatic content moderation is crucial to ensuring safety in social media. Language Model-based classifiers are being increasingly adopted for this task, but it has been shown that they perpetuate racial and social biases. Even if several resources and benchmark corpora have been developed to challenge this issue, measuring the fairness of models in content moderation remains an open issue. In this work, we present an unsupervised approach that benchmarks models on the basis of their uncertainty in classifying messages annotated by people belonging to vulnerable groups. We use uncertainty, computed by means of the conformal prediction technique, as a proxy to analyze the bias of 11 models against women and non-white annotators and observe to what extent it diverges from metrics based on performance, such as the F1 score. The results show that some pre-trained models predict with high accuracy the labels coming from minority groups, even if the confidence in their prediction is low. Therefore, by measuring the confidence of models, we are able to see which groups of annotators are better represented in pre-trained models and lead the debiasing process of these models before their effective use.