# Handling Bias in Toxic Speech Detection: A Survey

arXiv: https://arxiv.org/abs/2202.00126 · HTML: https://mlsanity.com/paper/2202.00126/

## Key facts

- Cited by 61 papers in the map (data through 14 Oct 2026).
- Builds on 61 of its 131 references that are in the map.
- First posted on arXiv 26 Jan 2022.

## Builds on (most cited first, 30 of 61)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [Equality of Opportunity in Supervised Learning](https://mlsanity.com/paper/1610.02413/) (2016)
- [Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings](https://mlsanity.com/paper/1607.06520/) (2016)
- [Ethical and social risks of harm from Language Models](https://mlsanity.com/paper/2112.04359/) (2021)
- [Language (Technology) is Power: A Critical Survey of "Bias" in NLP](https://mlsanity.com/paper/2005.14050/) (2020)
- [Semantics derived automatically from language corpora contain human-like biases](https://mlsanity.com/paper/1608.07187/) (2016)
- [Automated Hate Speech Detection and the Problem of Offensive Language](https://mlsanity.com/paper/1703.04009/) (2017)
- [Fair prediction with disparate impact: A study of bias in recidivism prediction instruments](https://mlsanity.com/paper/1610.07524/) (2016)
- [Gender Bias in Coreference Resolution: Evaluation and Debiasing Methods](https://mlsanity.com/paper/1804.06876/) (2018)
- [The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memes](https://mlsanity.com/paper/2005.04790/) (2020)
- [Dataset Cartography: Mapping and Diagnosing Datasets with Training Dynamics](https://mlsanity.com/paper/2009.10795/) (2020)
- [Nuanced Metrics for Measuring Unintended Bias with Real Data for Text Classification](https://mlsanity.com/paper/1903.04561/) (2019)
- [On Measuring Social Biases in Sentence Encoders](https://mlsanity.com/paper/1903.10561/) (2019)
- [BERTweet: A pre-trained language model for English Tweets](https://mlsanity.com/paper/2005.10200/) (2020)
- [Fairness in Criminal Justice Risk Assessments: The State of the Art](https://mlsanity.com/paper/1703.09207/) (2017)
- [Mitigating Gender Bias in Natural Language Processing: Literature Review](https://mlsanity.com/paper/1906.08976/) (2019)
- [Don't Take the Easy Way Out: Ensemble Based Methods for Avoiding Known Dataset Biases](https://mlsanity.com/paper/1909.03683/) (2019)
- [Large Scale Crowdsourcing and Characterization of Twitter Abusive Behavior](https://mlsanity.com/paper/1802.00393/) (2018)
- [Towards Fairness in Visual Recognition: Effective Strategies for Bias Mitigation](https://mlsanity.com/paper/1911.11834/) (2019)
- [Latent Hatred: A Benchmark for Understanding Implicit Hate Speech](https://mlsanity.com/paper/2109.05322/) (2021)
- [Ex Machina: Personal Attacks Seen at Scale](https://mlsanity.com/paper/1610.08914/) (2016)
- [Racial Bias in Hate Speech and Abusive Language Detection Datasets](https://mlsanity.com/paper/1905.12516/) (2019)
- [Hate Speech Dataset from a White Supremacy Forum](https://mlsanity.com/paper/1809.04444/) (2018)
- [Demographic Dialectal Variation in Social Media: A Case Study of African-American English](https://mlsanity.com/paper/1608.08868/) (2016)
- [Towards Debiasing Sentence Representations](https://mlsanity.com/paper/2007.08100/) (2020)
- [Challenges in Detoxifying Language Models](https://mlsanity.com/paper/2109.07445/) (2021)
- [Adversarial Filters of Dataset Biases](https://mlsanity.com/paper/2002.04108/) (2020)
- [Reducing Gender Bias in Abusive Language Detection](https://mlsanity.com/paper/1808.07231/) (2018)
- [Towards Debiasing Fact Verification Models](https://mlsanity.com/paper/1908.05267/) (2019)

## Built on by (most cited first, 30 of 61)

- [Scaling Instruction-Finetuned Language Models](https://mlsanity.com/paper/2210.11416/) (2022)
- [PaLM 2 Technical Report](https://mlsanity.com/paper/2305.10403/) (2023)
- [Representation Bias in Data: A Survey on Identification and Resolution Techniques](https://mlsanity.com/paper/2203.11852/) (2022)
- [Human and LLM Biases in Hate Speech Annotations: A Socio-Demographic Analysis of Annotators and Targets](https://mlsanity.com/paper/2410.07991/) (2024)
- [Disentangling Perceptions of Offensiveness: Cultural and Moral Correlates](https://mlsanity.com/paper/2312.06861/) (2023)
- [Recent Advances in Hate Speech Moderation: Multimodality and the Role of Large Models](https://mlsanity.com/paper/2401.16727/) (2024)
- [Hate Personified: Investigating the role of LLMs in content moderation](https://mlsanity.com/paper/2410.02657/) (2024)
- [NLP Systems That Can't Tell Use from Mention Censor Counterspeech, but Teaching the Distinction Helps](https://mlsanity.com/paper/2404.01651/) (2024)
- [LifeTox: Unveiling Implicit Toxicity in Life Advice](https://mlsanity.com/paper/2311.09585/) (2023)
- [Toxic Memes: A Survey of Computational Perspectives on the Detection and Explanation of Meme Toxicities](https://mlsanity.com/paper/2406.07353/) (2024)
- [Lost in Moderation: How Commercial Content Moderation APIs Over- and Under-Moderate Group-Targeted Hate Speech and Linguistic Variations](https://mlsanity.com/paper/2503.01623/) (2025)
- [CoSyn: Detecting Implicit Hate Speech in Online Conversations Using a Context Synergized Hyperbolic Network](https://mlsanity.com/paper/2303.03387/) (2023)
- [FUGNN: Harmonizing Fairness and Utility in Graph Neural Networks](https://mlsanity.com/paper/2405.17034/) (2024)
- [Tox-BART: Leveraging Toxicity Attributes for Explanation Generation of Implicit Hate Speech](https://mlsanity.com/paper/2406.03953/) (2024)
- [Don't Blame the Data, Blame the Model: Understanding Noise and Bias When Learning from Subjective Annotations](https://mlsanity.com/paper/2403.04085/) (2024)
- ["Cold, Calculated, and Condescending": How AI Identifies and Explains Ableism Compared to Disabled People](https://mlsanity.com/paper/2410.03448/) (2024)
- [Exploring Multimodal Challenges in Toxic Chinese Detection: Taxonomy, Benchmark, and Findings](https://mlsanity.com/paper/2505.24341/) (2025)
- [Causality Guided Disentanglement for Cross-Platform Hate Speech Detection](https://mlsanity.com/paper/2308.02080/) (2023)
- [Guardians and Offenders: A Survey on Harmful Content Generation and Safety Mitigation of LLM](https://mlsanity.com/paper/2508.05775/) (2025)
- [BiasTestGPT: Using ChatGPT for Social Bias Testing of Language Models](https://mlsanity.com/paper/2302.07371/) (2023)
- [Hate Speech Detection with Generalizable Target-aware Fairness](https://mlsanity.com/paper/2406.00046/) (2024)
- [Your spouse needs professional help: Determining the Contextual Appropriateness of Messages through Modeling Social Relationships](https://mlsanity.com/paper/2307.02763/) (2023)
- [BAN-PL: a Novel Polish Dataset of Banned Harmful and Offensive Content from Wykop.pl web service](https://mlsanity.com/paper/2308.10592/) (2023)
- [Lightweight Toxicity Detection in Spoken Language: A Transformer-based Approach for Edge Devices](https://mlsanity.com/paper/2304.11408/) (2023)
- [ToxicTone: A Mandarin Audio Dataset Annotated for Toxicity and Toxic Utterance Tonality](https://mlsanity.com/paper/2505.15773/) (2025)
- [A Taxonomy of Rater Disagreements: Surveying Challenges & Opportunities from the Perspective of Annotating Online Toxicity](https://mlsanity.com/paper/2311.04345/) (2023)
- [Exploring Human-LLM Conversations: Mental Models and the Originator of Toxicity](https://mlsanity.com/paper/2407.05977/) (2024)
- [A Review of Challenges in Machine Learning based Automated Hate Speech Detection](https://mlsanity.com/paper/2209.05294/) (2022)
- [Let Guidelines Guide You: A Prescriptive Guideline-Centered Data Annotation Methodology](https://mlsanity.com/paper/2406.14099/) (2024)
- [Watching the Watchers: A Comparative Fairness Audit of Cloud-based Content Moderation Services](https://mlsanity.com/paper/2406.14154/) (2024)

## Abstract (arXiv)

Detecting online toxicity has always been a challenge due to its inherent subjectivity. Factors such as the context, geography, socio-political climate, and background of the producers and consumers of the posts play a crucial role in determining if the content can be flagged as toxic. Adoption of automated toxicity detection models in production can thus lead to a sidelining of the various groups they aim to help in the first place. It has piqued researchers' interest in examining unintended biases and their mitigation. Due to the nascent and multi-faceted nature of the work, complete literature is chaotic in its terminologies, techniques, and findings. In this paper, we put together a systematic study of the limitations and challenges of existing methods for mitigating bias in toxicity detection. We look closely at proposed methods for evaluating and mitigating bias in toxic speech detection. To examine the limitations of existing methods, we also conduct a case study to introduce the concept of bias shift due to knowledge-based bias mitigation. The survey concludes with an overview of the critical challenges, research gaps, and future directions. While reducing toxicity on online platforms continues to be an active area of research, a systematic study of various biases and their mitigation strategies will help the research community produce robust and fair models.