# Watching the Watchers: A Comparative Fairness Audit of Cloud-based Content Moderation Services

arXiv: https://arxiv.org/abs/2406.14154 · HTML: https://mlsanity.com/paper/2406.14154/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 28 references that are in the map.
- First posted on arXiv 20 Jun 2024.

## Builds on (most cited first, 15 of 15)

- [The Woman Worked as a Babysitter: On Biases in Language Generation](https://mlsanity.com/paper/1909.01326/) (2019)
- [ToxiGen: A Large-Scale Machine-Generated Dataset for Adversarial and Implicit Hate Speech Detection](https://mlsanity.com/paper/2203.09509/) (2022)
- [HateXplain: A Benchmark Dataset for Explainable Hate Speech Detection](https://mlsanity.com/paper/2012.10289/) (2020)
- [Nuanced Metrics for Measuring Unintended Bias with Real Data for Text Classification](https://mlsanity.com/paper/1903.04561/) (2019)
- [Social Bias Frames: Reasoning about Social and Power Implications of Language](https://mlsanity.com/paper/1911.03891/) (2019)
- [Latent Hatred: A Benchmark for Understanding Implicit Hate Speech](https://mlsanity.com/paper/2109.05322/) (2021)
- [Counterfactual Fairness in Text Classification through Robustness](https://mlsanity.com/paper/1809.10610/) (2018)
- [Black-Box Access is Insufficient for Rigorous AI Audits](https://mlsanity.com/paper/2401.14446/) (2024)
- [Perturbation Sensitivity Analysis to Detect Unintended Model Biases](https://mlsanity.com/paper/1910.04210/) (2019)
- [Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance](https://mlsanity.com/paper/2206.04737/) (2022)
- [Handling Bias in Toxic Speech Detection: A Survey](https://mlsanity.com/paper/2202.00126/) (2022)
- [AI auditing: The Broken Bus on the Road to AI Accountability](https://mlsanity.com/paper/2401.14462/) (2024)
- [How Hate Speech Varies by Target Identity: A Computational Analysis](https://mlsanity.com/paper/2210.10839/) (2022)
- [Proactive Moderation of Online Discussions: Existing Practices and the Potential for Algorithmic Support](https://mlsanity.com/paper/2211.16525/) (2022)
- [Generative AI for Hate Speech Detection: Evaluation and Findings](https://mlsanity.com/paper/2311.09993/) (2023)

## Built on by (most cited first, 3 of 3)

- [Cracking the Code: Enhancing Implicit Hate Speech Detection through Coding Classification](https://mlsanity.com/paper/2506.04693/) (2025)
- [Mod-Guide: An LLM-based Content Moderation Feedback System to Address Insensitive Speech toward Indigenous Ethnic and Religious Minority Communities](https://mlsanity.com/paper/2606.13397/) (2026)
- [Characterizing Bluesky Content Moderation Service: From Automation of Service to Landscape of Harms](https://mlsanity.com/paper/2609.11373/) (2026)

## Abstract (arXiv)

Online platforms face the challenge of moderating an ever-increasing volume of content, including harmful hate speech. In the absence of clear legal definitions and a lack of transparency regarding the role of algorithms in shaping decisions on content moderation, there is a critical need for external accountability. Our study contributes to filling this gap by systematically evaluating four leading cloud-based content moderation services through a third-party audit, highlighting issues such as biases against minorities and vulnerable groups that may arise through over-reliance on these services. Using a black-box audit approach and four benchmark data sets, we measure performance in explicit and implicit hate speech detection as well as counterfactual fairness through perturbation sensitivity analysis and present disparities in performance for certain target identity groups and data sets. Our analysis reveals that all services had difficulties detecting implicit hate speech, which relies on more subtle and codified messages. Moreover, our results point to the need to remove group-specific bias. It seems that biases towards some groups, such as Women, have been mostly rectified, while biases towards other groups, such as LGBTQ+ and PoC remain.