mlsanityData to 7 Oct 2026

Handling Bias in Toxic Speech Detection: A Survey

arXiv 2202.00126 ↗First posted 26 Jan 2022cs.SI cs.CY cs.LGcited by 60

Not on the map. Why?

Abstract (arXiv)

Detecting online toxicity has always been a challenge due to its inherent subjectivity. Factors such as the context, geography, socio-political climate, and background of the producers and consumers of the posts play a crucial role in determining if the content can be flagged as toxic. Adoption of automated toxicity detection models in production can thus lead to a sidelining of the various groups they aim to help in the first place. It has piqued researchers' interest in examining unintended biases and their mitigation. Due to the nascent and multi-faceted nature of the work, complete literature is chaotic in its terminologies, techniques, and findings. In this paper, we put together a systematic study of the limitations and challenges of existing methods for mitigating bias in toxicity detection. We look closely at proposed methods for evaluating and mitigating bias in toxic speech detection. To examine the limitations of existing methods, we also conduct a case study to introduce the concept of bias shift due to knowledge-based bias mitigation. The survey concludes with an overview of the critical challenges, research gaps, and future directions. While reducing toxicity on online platforms continues to be an active area of research, a systematic study of various biases and their mitigation strategies will help the research community produce robust and fair models.

Key facts. Cited by 60 papers in the map (data through 7 Oct 2026). Builds on 61 of its 131 references that are in the map. First posted on arXiv 26 Jan 2022.

Builds on

61 of 131 references.

References · 61

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,346
  3. 2016Equality of Opportunity in Supervised Learningcited by 2,000
  4. 2016Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddingscited by 1,304
  5. 2021Ethical and social risks of harm from Language Modelscited by 957
  6. 2020Language (Technology) is Power: A Critical Survey of "Bias" in NLPcited by 951
  7. 2016Semantics derived automatically from language corpora contain human-like biasescited by 894
  8. 2017Automated Hate Speech Detection and the Problem of Offensive Languagecited by 724
  9. 2016Fair prediction with disparate impact: A study of bias in recidivism prediction instrumentscited by 671
  10. 2018Gender Bias in Coreference Resolution: Evaluation and Debiasing Methodscited by 657
  11. 2020The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memescited by 485
  12. 2020Dataset Cartography: Mapping and Diagnosing Datasets with Training Dynamicscited by 423
38 more
  1. 2019On Measuring Social Biases in Sentence Encoderscited by 397
  2. 2019Nuanced Metrics for Measuring Unintended Bias with Real Data for Text Classificationcited by 396
  3. 2020BERTweet: A pre-trained language model for English Tweetscited by 346
  4. 2017Fairness in Criminal Justice Risk Assessments: The State of the Artcited by 315
  5. 2019Mitigating Gender Bias in Natural Language Processing: Literature Reviewcited by 287
  6. 2019Don't Take the Easy Way Out: Ensemble Based Methods for Avoiding Known Dataset Biasescited by 279
  7. 2018Large Scale Crowdsourcing and Characterization of Twitter Abusive Behaviorcited by 265
  8. 2019Towards Fairness in Visual Recognition: Effective Strategies for Bias Mitigationcited by 252
  9. 2021Latent Hatred: A Benchmark for Understanding Implicit Hate Speechcited by 216
  10. 2016Ex Machina: Personal Attacks Seen at Scalecited by 208
  11. 2019Racial Bias in Hate Speech and Abusive Language Detection Datasetscited by 208
  12. 2018Hate Speech Dataset from a White Supremacy Forumcited by 196
  13. 2016Demographic Dialectal Variation in Social Media: A Case Study of African-American Englishcited by 194
  14. 2020Towards Debiasing Sentence Representationscited by 170
  15. 2021Challenges in Detoxifying Language Modelscited by 169
  16. 2020Adversarial Filters of Dataset Biasescited by 159
  17. 2018Reducing Gender Bias in Abusive Language Detectioncited by 152
  18. 2019Towards Debiasing Fact Verification Modelscited by 143
  19. 2021MultiVerS: Improving scientific claim verification with weak supervision and full-document contextcited by 140
  20. 2019Predictive Biases in Natural Language Processing Models: A Conceptual Framework and Overviewcited by 137
  21. 2021Harms of Gender Exclusivity and Challenges in Non-Binary Representation in Language Technologiescited by 128
  22. 2017Measuring the Reliability of Hate Speech Annotations: The Case of the European Refugee Crisiscited by 95
  23. 2021Challenges in Automated Debiasing for Toxic Language Detectioncited by 95
  24. 2020Hate Speech Detection and Racial Bias Mitigation in Social Media based on BERT modelcited by 87
  25. 2021MOMENTA: A Multimodal Framework for Detecting Harmful Memes and Their Targetscited by 85
  26. 2020Contextualizing Hate Speech Classifiers with Post-hoc Explanationcited by 84
  27. 2021Detecting Harmful Memes and Their Targetscited by 77
  28. 2021Towards generalisable hate speech detection: a review on obstacles and solutionscited by 66
  29. 2020Demoting Racial Bias in Hate Speech Detectioncited by 62
  30. 2018Spread of hate speech in online social mediacited by 59
  31. 2020Multilingual Twitter Corpus and Baselines for Evaluating Demographic Bias in Hate Speech Recognitioncited by 43
  32. 2020Demographics Should Not Be the Reason of Toxicity: Mitigating Discrimination in Text Classifications with Instance Weightingcited by 41
  33. 2020Stereotypical Bias Removal for Hate Speech Detection Task using Knowledge-based Generalizationscited by 40
  34. 2021Algorithmic Amplification of Politics on Twittercited by 38
  35. 2021Detecting Cross-Geographic Biases in Toxicity Modeling on Social Mediacited by 31
  36. 2018Leveraging Intra-User and Inter-User Representation Learning for Automated Hate Speech Detectioncited by 25
  37. 2021Measuring Model Biases in the Absence of Ground Truthcited by 24
  38. 2020Intersectional Bias in Hate Speech and Abusive Language Datasetscited by 22

50 most cited of 61 All 61 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 3; 2023: 12; 2024: 21; 2025: 13; 2026: 11.

60 papers.

Sort:

By area:

  1. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  2. 2023PaLM 2 Technical Reportcited by 1,154
  3. 2022Representation Bias in Data: A Survey on Identification and Resolution Techniquescited by 37
  4. 2024Human and LLM Biases in Hate Speech Annotations: A Socio-Demographic Analysis of Annotators and Targetscited by 31
  5. 2023Disentangling Perceptions of Offensiveness: Cultural and Moral Correlatescited by 26
  6. 2024Recent Advances in Hate Speech Moderation: Multimodality and the Role of Large Modelscited by 26
  7. 2024Hate Personified: Investigating the role of LLMs in content moderationcited by 23
  8. 2024NLP Systems That Can't Tell Use from Mention Censor Counterspeech, but Teaching the Distinction Helpscited by 19
  9. 2023LifeTox: Unveiling Implicit Toxicity in Life Advicecited by 14
  10. 2024Toxic Memes: A Survey of Computational Perspectives on the Detection and Explanation of Meme Toxicitiescited by 13
  11. 2023CoSyn: Detecting Implicit Hate Speech in Online Conversations Using a Context Synergized Hyperbolic Networkcited by 12
  12. 2025Lost in Moderation: How Commercial Content Moderation APIs Over- and Under-Moderate Group-Targeted Hate Speech and Linguistic Variationscited by 12
38 more
  1. 2024FUGNN: Harmonizing Fairness and Utility in Graph Neural Networkscited by 11
  2. 2024Tox-BART: Leveraging Toxicity Attributes for Explanation Generation of Implicit Hate Speechcited by 11
  3. 2024Don't Blame the Data, Blame the Model: Understanding Noise and Bias When Learning from Subjective Annotationscited by 10
  4. 2025Exploring Multimodal Challenges in Toxic Chinese Detection: Taxonomy, Benchmark, and Findingscited by 10
  5. 2023Causality Guided Disentanglement for Cross-Platform Hate Speech Detectioncited by 9
  6. 2024"Cold, Calculated, and Condescending": How AI Identifies and Explains Ableism Compared to Disabled Peoplecited by 9
  7. 2023BiasTestGPT: Using ChatGPT for Social Bias Testing of Language Modelscited by 8
  8. 2025Guardians and Offenders: A Survey on Harmful Content Generation and Safety Mitigation of LLMcited by 8
  9. 2024Hate Speech Detection with Generalizable Target-aware Fairnesscited by 7
  10. 2023Your spouse needs professional help: Determining the Contextual Appropriateness of Messages through Modeling Social Relationshipscited by 6
  11. 2023BAN-PL: a Novel Polish Dataset of Banned Harmful and Offensive Content from Wykop.pl web servicecited by 6
  12. 2025ToxicTone: A Mandarin Audio Dataset Annotated for Toxicity and Toxic Utterance Tonalitycited by 5
  13. 2023A Taxonomy of Rater Disagreements: Surveying Challenges & Opportunities from the Perspective of Annotating Online Toxicitycited by 4
  14. 2024Exploring Human-LLM Conversations: Mental Models and the Originator of Toxicitycited by 4
  15. 2022A Review of Challenges in Machine Learning based Automated Hate Speech Detectioncited by 3
  16. 2024Let Guidelines Guide You: A Prescriptive Guideline-Centered Data Annotation Methodologycited by 3
  17. 2024Watching the Watchers: A Comparative Fairness Audit of Cloud-based Content Moderation Servicescited by 3
  18. 2024Leveraging Machine Learning to Identify Gendered Stereotypes and Body Image Concerns on Diet and Fitness Online Forumscited by 3
  19. 2025DICE: A Framework for Dimensional and Contextual Evaluation of Language Modelscited by 3
  20. 2025Toxicity in Online Platforms and AI Systems: A Survey of Needs, Challenges, Mitigations, and Future Directionscited by 3
  21. 2023Mitigating Bias in Conversations: A Hate Speech Classifier and Debiaser with Promptscited by 2
  22. 2023Focal Inferential Infusion Coupled with Tractable Density Discrimination for Implicit Hate Detectioncited by 2
  23. 2024Cross-lingual Offensive Language Detection: A Systematic Review of Datasets, Transfer Approaches and Challengescited by 2
  24. 2024Algorithmic Fairness: A Tolerance Perspectivecited by 2
  25. 2024On the Role of Speech Data in Reducing Toxicity Detection Biascited by 2
  26. 2024A Federated Approach to Few-Shot Hate Speech Detection for Marginalized Communitiescited by 2
  27. 2025Counterspeech the ultimate shield! Multi-Conditioned Counterspeech Generation through Attributed Prefix Learningcited by 2
  28. 2025Camellia: Benchmarking Cultural Biases in LLMs for Asian Languagescited by 2
  29. 2026When to Invoke: Refining LLM Fairness with Toxicity Assessmentcited by 2
  30. 2023On the definition of toxicity in NLPcited by 1
  31. 2024Legally Binding but Unfair? Towards Assessing Fairness of Privacy Policiescited by 1
  32. 2024Dataverse: Open-Source ETL (Extract, Transform, Load) Pipeline for Large Language Modelscited by 1
  33. 2026Analyzing Bias in False Refusal Behavior of Large Language Models for Hate Speech Detoxificationcited by 1
  34. 2026Bye Bye Perspective API: Lessons for Building and Governing Measurement Infrastructurecited by 1
  35. 2024Mitigating Biases to Embrace Diversity: A Comprehensive Annotation Benchmark for Toxic Languagecited by 0
  36. 2025Toxicity Detection Should Measure Contextual Harm, Not Text-Intrinsic Badnesscited by 0
  37. 2025Tackling Social Bias against the Poor: A Dataset and Taxonomy on Aporophobiacited by 0
  38. 2025A Survey on Stereotype Detection in Natural Language Processingcited by 0

50 most cited of 60 All 60 (JSON)