# Towards Equal Gender Representation in the Annotations of Toxic Language Detection

arXiv: https://arxiv.org/abs/2106.02183 · HTML: https://mlsanity.com/paper/2106.02183/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 22 references that are in the map.
- First posted on arXiv 4 Jun 2021.

## Builds on (most cited first, 15 of 15)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Axiomatic Attribution for Deep Networks](https://mlsanity.com/paper/1703.01365/) (2017)
- [Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings](https://mlsanity.com/paper/1607.06520/) (2016)
- [Language (Technology) is Power: A Critical Survey of "Bias" in NLP](https://mlsanity.com/paper/2005.14050/) (2020)
- [Gender Bias in Coreference Resolution: Evaluation and Debiasing Methods](https://mlsanity.com/paper/1804.06876/) (2018)
- [Mitigating Gender Bias in Natural Language Processing: Literature Review](https://mlsanity.com/paper/1906.08976/) (2019)
- [Measuring Bias in Contextualized Word Representations](https://mlsanity.com/paper/1906.07337/) (2019)
- [Ex Machina: Personal Attacks Seen at Scale](https://mlsanity.com/paper/1610.08914/) (2016)
- [Black is to Criminal as Caucasian is to Police: Detecting and Removing Multiclass Bias in Word Embeddings](https://mlsanity.com/paper/1904.04047/) (2019)
- [Toxicity Detection: Does Context Really Matter?](https://mlsanity.com/paper/2006.00998/) (2020)
- [Like trainer, like bot? Inheritance of bias in algorithmic content moderation](https://mlsanity.com/paper/1707.01477/) (2017)
- [Deep Learning for User Comment Moderation](https://mlsanity.com/paper/1705.09993/) (2017)
- [Neural Character-based Composition Models for Abuse Detection](https://mlsanity.com/paper/1809.00378/) (2018)
- [Classifying Constructive Comments](https://mlsanity.com/paper/2004.05476/) (2020)
- [Debiasing Personal Identities in Toxicity Classification](https://mlsanity.com/paper/1908.05757/) (2019)

## Built on by (most cited first, 9 of 9)

- [Sensitivity, Performance, Robustness: Deconstructing the Effect of Sociodemographic Prompting](https://mlsanity.com/paper/2309.07034/) (2023)
- [The Ecological Fallacy in Annotation: Modelling Human Label Variation goes beyond Sociodemographics](https://mlsanity.com/paper/2306.11559/) (2023)
- [Gender Bias in BERT -- Measuring and Analysing Biases through Sentiment Rating in a Realistic Downstream Classification Task](https://mlsanity.com/paper/2306.15298/) (2023)
- [Analyzing Hate Speech Data along Racial, Gender and Intersectional Axes](https://mlsanity.com/paper/2205.06621/) (2022)
- [Assessing the Reliability of LLMs Annotations in the Context of Demographic Bias and Model Explanation](https://mlsanity.com/paper/2507.13138/) (2025)
- [Ethics Whitepaper: Whitepaper on Ethical Research into Large Language Models](https://mlsanity.com/paper/2410.19812/) (2024)
- [An Approach to Ensure Fairness in News Articles](https://mlsanity.com/paper/2207.03938/) (2022)
- [Don't Take it Personally: Analyzing Gender and Age Differences in Ratings of Online Humor](https://mlsanity.com/paper/2208.10898/) (2022)
- [A Study on Bias Detection and Classification in Natural Language Processing](https://mlsanity.com/paper/2408.07479/) (2024)

## Abstract (arXiv)

Classifiers tend to propagate biases present in the data on which they are trained. Hence, it is important to understand how the demographic identities of the annotators of comments affect the fairness of the resulting model. In this paper, we focus on the differences in the ways men and women annotate comments for toxicity, investigating how these differences result in models that amplify the opinions of male annotators. We find that the BERT model as-sociates toxic comments containing offensive words with male annotators, causing the model to predict 67.7% of toxic comments as having been annotated by men. We show that this disparity between gender predictions can be mitigated by removing offensive words and highly toxic comments from the training data. We then apply the learned associations between gender and language to toxic language classifiers, finding that models trained exclusively on female-annotated data perform 1.8% better than those trained solely on male-annotated data and that training models on data after removing all offensive words reduces bias in the model by 55.5% while increasing the sensitivity by 0.4%.