# Silencing Empowerment, Allowing Bigotry: Auditing the Moderation of Hate Speech on Twitch

arXiv: https://arxiv.org/abs/2506.07667 · HTML: https://mlsanity.com/paper/2506.07667/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 16 of its 69 references that are in the map.
- First posted on arXiv 9 Jun 2025.

## Builds on (most cited first, 16 of 16)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Stealing Machine Learning Models via Prediction APIs](https://mlsanity.com/paper/1609.02943/) (2016)
- [ToxiGen: A Large-Scale Machine-Generated Dataset for Adversarial and Implicit Hate Speech Detection](https://mlsanity.com/paper/2203.09509/) (2022)
- [Social Bias Frames: Reasoning about Social and Power Implications of Language](https://mlsanity.com/paper/1911.03891/) (2019)
- [Learning from the Worst: Dynamically Generated Datasets to Improve Online Hate Detection](https://mlsanity.com/paper/2012.15761/) (2020)
- [Latent Hatred: A Benchmark for Understanding Implicit Hate Speech](https://mlsanity.com/paper/2109.05322/) (2021)
- [Detecting Harmful Content On Online Platforms: What Platforms Need Vs. Where Research Efforts Go](https://mlsanity.com/paper/2103.00153/) (2021)
- [From Transparency to Accountability and Back: A Discussion of Access and Evidence in AI Auditing](https://mlsanity.com/paper/2410.04772/) (2024)
- [Lost in Moderation: How Commercial Content Moderation APIs Over- and Under-Moderate Group-Targeted Hate Speech and Linguistic Variations](https://mlsanity.com/paper/2503.01623/) (2025)
- ["Community Guidelines Make this the Best Party on the Internet": An In-Depth Study of Online Platforms' Content Moderation Policies](https://mlsanity.com/paper/2405.05225/) (2024)
- [Automated Content Moderation Increases Adherence to Community Guidelines](https://mlsanity.com/paper/2210.10454/) (2022)
- [Hate Raids on Twitch: Echoes of the Past, New Modalities, and Implications for Platform Governance](https://mlsanity.com/paper/2301.03946/) (2023)
- [Trajectories of Blocked Community Members: Redemption, Recidivism and Departure](https://mlsanity.com/paper/1902.08628/) (2019)
- [Content Moderation Justice and Fairness on Social Media: Comparisons Across Different Contexts and Platforms](https://mlsanity.com/paper/2403.06034/) (2024)
- [The Uli Dataset: An Exercise in Experience Led Annotation of oGBV](https://mlsanity.com/paper/2311.09086/) (2023)
- [Software-Supported Audits of Decision-Making Systems: Testing Google and Facebook's Political Advertising Policies](https://mlsanity.com/paper/2103.00064/) (2021)

## Abstract (arXiv)

To meet the demands of content moderation, online platforms have resorted to automated systems. Newer forms of real-time engagement(textite.g., users commenting on live streams) on platforms like Twitch exert additional pressures on the latency expected of such moderation systems. Despite their prevalence, relatively little is known about the effectiveness of these systems. In this paper, we conduct an audit of Twitch's automated moderation tool (textttAutoMod) to investigate its effectiveness in flagging hateful content. For our audit, we create streaming accounts to act as siloed test beds, and interface with the live chat using Twitch's APIs to send over 107,000 comments collated from 4 datasets. We measure textttAutoMod's accuracy in flagging blatantly hateful content containing misogyny, racism, ableism and homophobia. Our experiments reveal that a large fraction of hateful messages, up to 94% on some datasets, textitbypass moderation. Contextual addition of slurs to these messages results in 100% removal, revealing textttAutoMod's reliance on slurs as a moderation signal. We also find that contrary to Twitch's community guidelines, textttAutoMod blocks up to 89.5% of benign examples that use sensitive words in pedagogical or empowering contexts. Overall, our audit points to large gaps in textttAutoMod's capabilities and underscores the importance for such systems to understand context effectively.