# Toxicity Detection is NOT all you Need: Measuring the Gaps to Supporting Volunteer Content Moderators

arXiv: https://arxiv.org/abs/2311.07879 · HTML: https://mlsanity.com/paper/2311.07879/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 30 references that are in the map.
- First posted on arXiv 14 Nov 2023.

## Builds on (most cited first, 11 of 11)

- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [A Holistic Approach to Undesired Content Detection in the Real World](https://mlsanity.com/paper/2208.03274/) (2022)
- [Predicting the Type and Target of Offensive Posts in Social Media](https://mlsanity.com/paper/1902.09666/) (2019)
- [A New Generation of Perspective API: Efficient Multilingual Character-level Transformers](https://mlsanity.com/paper/2202.11176/) (2022)
- [Toxicity Detection: Does Context Really Matter?](https://mlsanity.com/paper/2006.00998/) (2020)
- [One-step and Two-step Classification for Abusive Language Detection on Twitter](https://mlsanity.com/paper/1706.01206/) (2017)
- [Identifying Machine-Paraphrased Plagiarism](https://mlsanity.com/paper/2103.11909/) (2021)
- [Hate Raids on Twitch: Echoes of the Past, New Modalities, and Implications for Platform Governance](https://mlsanity.com/paper/2301.03946/) (2023)
- [Measuring the Monetary Value of Online Volunteer Work](https://mlsanity.com/paper/2205.14528/) (2022)
- [Moderation Challenges in Voice-based Online Communities on Discord](https://mlsanity.com/paper/2101.05258/) (2021)
- [Surviving an "Eternal September" - How an Online Community Managed a Surge of Newcomers](https://mlsanity.com/paper/1605.08841/) (2016)

## Built on by (most cited first, 10 of 10)

- [Content Moderation by LLM: From Accuracy to Legitimacy](https://mlsanity.com/paper/2409.03219/) (2024)
- [Plurals: A System for Guiding LLMs Via Simulated Social Ensembles](https://mlsanity.com/paper/2409.17213/) (2024)
- [Guardians and Offenders: A Survey on Harmful Content Generation and Safety Mitigation of LLM](https://mlsanity.com/paper/2508.05775/) (2025)
- [Seeing Like an AI: How LLMs Apply (and Misapply) Wikipedia Neutrality Norms](https://mlsanity.com/paper/2407.04183/) (2024)
- [Unified Game Moderation: Soft-Prompting and LLM-Assisted Label Transfer for Resource-Efficient Toxicity Detection](https://mlsanity.com/paper/2506.06347/) (2025)
- [Chinese Toxic Language Mitigation via Sentiment Polarity Consistent Rewrites](https://mlsanity.com/paper/2505.15297/) (2025)
- [Predictively Combatting Toxicity in Health-related Online Discussions through Machine Learning](https://mlsanity.com/paper/2505.17068/) (2025)
- [SMARTER: A Data-efficient Framework to Improve Toxicity Detection with Explanation via Self-augmenting Large Language Models](https://mlsanity.com/paper/2509.15174/) (2025)
- [On the Sensitivity of Instruction-tuned LLMs to Harmful Sentences in Long Inputs](https://mlsanity.com/paper/2510.05864/) (2025)
- [VARM-Bench: Benchmarking Verifiable Structured Reasoning in Chinese Abusive Speech Moderation](https://mlsanity.com/paper/2608.15600/) (2026)

## Abstract (arXiv)

Extensive efforts in automated approaches for content moderation have been focused on developing models to identify toxic, offensive, and hateful content with the aim of lightening the load for moderators. Yet, it remains uncertain whether improvements on those tasks have truly addressed moderators' needs in accomplishing their work. In this paper, we surface gaps between past research efforts that have aimed to provide automation for aspects of content moderation and the needs of volunteer content moderators, regarding identifying violations of various moderation rules. To do so, we conduct a model review on Hugging Face to reveal the availability of models to cover various moderation rules and guidelines from three exemplar forums. We further put state-of-the-art LLMs to the test, evaluating how well these models perform in flagging violations of platform rules from one particular forum. Finally, we conduct a user survey study with volunteer moderators to gain insight into their perspectives on useful moderation models. Overall, we observe a non-trivial gap, as missing developed models and LLMs exhibit moderate to low performance on a significant portion of the rules. Moderators' reports provide guides for future work on developing moderation assistant models.