mlsanityData to 14 Oct 2026

Lost in Moderation: How Commercial Content Moderation APIs Over- and Under-Moderate Group-Targeted Hate Speech and Linguistic Variations

arXiv 2503.01623 ↗First posted 3 Mar 2025cs.HC cs.CLcited by 13

Not on the map. Why?

Abstract (arXiv)

Commercial content moderation APIs are marketed as scalable solutions to combat online hate speech. However, the reliance on these APIs risks both silencing legitimate speech, called over-moderation, and failing to protect online platforms from harmful speech, known as under-moderation. To assess such risks, this paper introduces a framework for auditing black-box NLP systems. Using the framework, we systematically evaluate five widely used commercial content moderation APIs. Analyzing five million queries based on four datasets, we find that APIs frequently rely on group identity terms, such as ``black'', to predict hate speech. While OpenAI's and Amazon's services perform slightly better, all providers under-moderate implicit hate speech, which uses codified messages, especially against LGBTQIA+ individuals. Simultaneously, they over-moderate counter-speech, reclaimed slurs and content related to Black, LGBTQIA+, Jewish, and Muslim people. We recommend that API providers offer better guidance on API implementation and threshold setting and more transparency on their APIs' limitations. Warning: This paper contains offensive and hateful terms and concepts. We have chosen to reproduce these terms for reasons of transparency.

Key facts. Cited by 13 papers in the map (data through 14 Oct 2026). Builds on 46 of its 111 references that are in the map. First posted on arXiv 3 Mar 2025.

Builds on

46 of 111 references.

References · 46

  1. 2017A Unified Approach to Interpreting Model Predictionscited by 6,555
  2. 2016Equality of Opportunity in Supervised Learningcited by 2,007
  3. 2018Model Cards for Model Reportingcited by 1,080
  4. 2020Language (Technology) is Power: A Critical Survey of "Bias" in NLPcited by 956
  5. 2023Bias and Fairness in Large Language Models: A Surveycited by 639
  6. 2019The Woman Worked as a Babysitter: On Biases in Language Generationcited by 541
  7. 2022ToxiGen: A Large-Scale Machine-Generated Dataset for Adversarial and Implicit Hate Speech Detectioncited by 422
  8. 2020HateXplain: A Benchmark Dataset for Explainable Hate Speech Detectioncited by 407
  9. 2019Nuanced Metrics for Measuring Unintended Bias with Real Data for Text Classificationcited by 404
  10. 2019Social Bias Frames: Reasoning about Social and Power Implications of Languagecited by 380
  11. 2019Interpretable machine learning: definitions, methods, and applicationscited by 305
  12. 2021Annotators with Attitudes: How Annotator Beliefs And Identities Bias Toxic Language Detectioncited by 264
34 more
  1. 2020Learning from the Worst: Dynamically Generated Datasets to Improve Online Hate Detectioncited by 221
  2. 2021Latent Hatred: A Benchmark for Understanding Implicit Hate Speechcited by 217
  3. 2020HateCheck: Functional Tests for Hate Speech Detection Modelscited by 216
  4. 2019Racial Bias in Hate Speech and Abusive Language Detection Datasetscited by 209
  5. 2021Trustworthy AI: From Principles to Practicescited by 207
  6. 2018Reducing Gender Bias in Abusive Language Detectioncited by 152
  7. 2022Towards Faithful Model Explanation in NLP: A Surveycited by 130
  8. 2024Black-Box Access is Insufficient for Rigorous AI Auditscited by 126
  9. 2017Understanding Abuse: A Typology of Abusive Language Detection Subtaskscited by 116
  10. 2022The Shapley Value in Machine Learningcited by 98
  11. 2021Challenges in Automated Debiasing for Toxic Language Detectioncited by 96
  12. 2020Hate Speech Detection and Racial Bias Mitigation in Social Media based on BERT modelcited by 87
  13. 2019Perturbation Sensitivity Analysis to Detect Unintended Model Biasescited by 74
  14. 2022Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governancecited by 67
  15. 2021Towards generalisable hate speech detection: a review on obstacles and solutionscited by 66
  16. 2020Demoting Racial Bias in Hate Speech Detectioncited by 62
  17. 2022Handling Bias in Toxic Speech Detection: A Surveycited by 61
  18. 2024AI auditing: The Broken Bus on the Road to AI Accountabilitycited by 43
  19. 2017Like trainer, like bot? Inheritance of bias in algorithmic content moderationcited by 35
  20. 2019Tackling Online Abuse: A Survey of Automated Abuse Detection Methodscited by 33
  21. 2022Hate Speech and Counter Speech Detection: Conversational Context Does Mattercited by 32
  22. 2021Detecting Cross-Geographic Biases in Toxicity Modeling on Social Mediacited by 31
  23. 2022How Hate Speech Varies by Target Identity: A Computational Analysiscited by 23
  24. 2021Problematic Machine Behavior: A Systematic Literature Review of Algorithm Auditscited by 21
  25. 2023Toxic Bias: Perspective API Misreads German as More Toxiccited by 21
  26. 2023Automated Ableism: An Exploration of Explicit Disability Biases in Sentiment and Toxicity Analysis Modelscited by 20
  27. 2024Counterspeakers' Perspectives: Unveiling Barriers and AI Needs in the Fight against Online Hatecited by 15
  28. 2024"Community Guidelines Make this the Best Party on the Internet": An In-Depth Study of Online Platforms' Content Moderation Policiescited by 11
  29. 2023BiasX: "Thinking Slow" in Toxic Content Moderation with Explanations of Implied Social Biasescited by 7
  30. 2024Addressing the regulatory gap: moving towards an EU AI audit ecosystem beyond the AI Act by including civil societycited by 7
  31. 2024Learning to Defer in Congested Systems: The AI-Human Interplaycited by 6
  32. 2023How toxic is antisemitism? Potentials and limitations of automated toxicity scoring for antisemitic online contentcited by 4
  33. 2024Content Moderation Justice and Fairness on Social Media: Comparisons Across Different Contexts and Platformscited by 4
  34. 2023Online Harassment in Majority Contexts: Examining Harms and Remedies across Countriescited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 4; 2026: 9.

13 papers.

Sort:

By area:

  1. 2025When Your Reviewer is an LLM: Biases, Divergence, and Prompt Injection Risks in Peer Reviewcited by 19
  2. 2026Audit Me If You Can: Query-Efficient Active Fairness Auditing of Black-Box LLMscited by 6
  3. 2025Probing Association Biases in LLM Moderation Over-Sensitivitycited by 3
  4. 2025Longitudinal Monitoring of LLM Content Moderation of Social Issuescited by 2
  5. 2026Human-Centred LLM Privacy Audits: Findings and Frictionscited by 1
  6. 2026Bye Bye Perspective API: Lessons for Building and Governing Measurement Infrastructurecited by 1
  7. 2025Silencing Empowerment, Allowing Bigotry: Auditing the Moderation of Hate Speech on Twitchcited by 0
  8. 2026Turning Trust to Transactions: Tracking Affiliate Marketing and FTC Compliance in YouTube's Influencer Economycited by 0
  9. 2026You Frame It: How Conceptual Representations Shape LLM Detection and Reasoning about Antisemitismcited by 0
  10. 2026Old Tricks, New Models: How Simple Image Transformations Break Modern AI-based Content Moderationcited by 0
  11. 2026Ctrl-F-Resist. Practices, Challenges, and Technical Needs of Civil Society Organizations Monitoring the Far-Right Onlinecited by 0
  12. 2026Characterizing Bluesky Content Moderation Service: From Automation of Service to Landscape of Harmscited by 0
1 more
  1. 2026Hard-Gate Candidacy in a Deployed Validator Suitecited by 0