# Watch Your Language: Investigating Content Moderation with Large Language Models

arXiv: https://arxiv.org/abs/2309.14517 · HTML: https://mlsanity.com/paper/2309.14517/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 74 references that are in the map.
- First posted on arXiv 25 Sep 2023.

## Builds on (most cited first, 20 of 20)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [ChatGPT Outperforms Crowd-Workers for Text-Annotation Tasks](https://mlsanity.com/paper/2303.15056/) (2023)
- [A Prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPT](https://mlsanity.com/paper/2302.11382/) (2023)
- [Self-Diagnosis and Self-Debiasing: A Proposal for Reducing Corpus-Based Bias in NLP](https://mlsanity.com/paper/2103.00453/) (2021)
- [How is ChatGPT's behavior changing over time?](https://mlsanity.com/paper/2307.09009/) (2023)
- [Latent Hatred: A Benchmark for Understanding Implicit Hate Speech](https://mlsanity.com/paper/2109.05322/) (2021)
- [ChatGPT-4 Outperforms Experts and Crowd Workers in Annotating Political Twitter Messages with Zero-Shot Learning](https://mlsanity.com/paper/2304.06588/) (2023)
- [Toxicity Detection: Does Context Really Matter?](https://mlsanity.com/paper/2006.00998/) (2020)
- [Conversations Gone Awry: Detecting Early Signs of Conversational Failure](https://mlsanity.com/paper/1805.05345/) (2018)
- [Human-AI Collaboration via Conditional Delegation: A Case Study of Content Moderation](https://mlsanity.com/paper/2204.11788/) (2022)
- [Designing Toxic Content Classification for a Diversity of Perspectives](https://mlsanity.com/paper/2106.04511/) (2021)
- [Detecting Hate Speech with GPT-3](https://mlsanity.com/paper/2103.12407/) (2021)
- ["HOT" ChatGPT: The promise of ChatGPT in detecting and discriminating hateful, offensive, and toxic comments on social media](https://mlsanity.com/paper/2304.10619/) (2023)
- [You Only Prompt Once: On the Capabilities of Prompt Learning on Large Language Models to Tackle Toxic Content](https://mlsanity.com/paper/2308.05596/) (2023)
- [The Structure of Toxic Conversations on Twitter](https://mlsanity.com/paper/2105.11596/) (2021)
- [Mitigating Racial Biases in Toxic Language Detection with an Equity-Based Ensemble Framework](https://mlsanity.com/paper/2109.13137/) (2021)
- [A Trade-off-centered Framework of Content Moderation](https://mlsanity.com/paper/2206.03450/) (2022)
- [Hate Raids on Twitch: Echoes of the Past, New Modalities, and Implications for Platform Governance](https://mlsanity.com/paper/2301.03946/) (2023)
- [Twits, Toxic Tweets, and Tribal Tendencies: Trends in Politically Polarized Posts on Twitter](https://mlsanity.com/paper/2307.10349/) (2023)

## Built on by (most cited first, 14 of 14)

- [GuardReasoner: Towards Reasoning-based LLM Safeguards](https://mlsanity.com/paper/2501.18492/) (2025)
- [Harmful Fine-tuning Attacks and Defenses for Large Language Models: A Survey](https://mlsanity.com/paper/2409.18169/) (2024)
- [What Does the Bot Say? Opportunities and Risks of Large Language Models in Social Media Bot Detection](https://mlsanity.com/paper/2402.00371/) (2024)
- [Image-Based Geolocation Using Large Vision-Language Models](https://mlsanity.com/paper/2408.09474/) (2024)
- [STAND-Guard: A Small Task-Adaptive Content Moderation Model](https://mlsanity.com/paper/2411.05214/) (2024)
- [Self-Destructive Language Model](https://mlsanity.com/paper/2505.12186/) (2025)
- [APT-Pipe: A Prompt-Tuning Tool for Social Data Annotation using ChatGPT](https://mlsanity.com/paper/2402.01697/) (2024)
- [Evaluating the efficacy of LLM Safety Solutions : The Palit Benchmark Dataset](https://mlsanity.com/paper/2505.13028/) (2025)
- [Beyond Refusal: Probing the Limits of Agentic Self-Correction for Semantic Sensitive Information](https://mlsanity.com/paper/2602.21496/) (2026)
- [Efficacy of Utilizing Large Language Models to Detect Public Threat Posted Online](https://mlsanity.com/paper/2401.02974/) (2023)
- [The Unappreciated Role of Intent in Algorithmic Moderation of Social Media Content](https://mlsanity.com/paper/2405.11030/) (2024)
- [Assessing and Refining ChatGPT's Performance in Identifying Targeting and Inappropriate Language: A Comparative Study](https://mlsanity.com/paper/2505.21710/) (2025)
- [Mind the Gap: Pitfalls of LLM Alignment with Asian Public Opinion](https://mlsanity.com/paper/2603.06264/) (2026)
- [Compass: Continuously Aligning Social Media Feeds via In-Situ Reflections](https://mlsanity.com/paper/2608.04274/) (2026)

## Abstract (arXiv)

Large language models (LLMs) have exploded in popularity due to their ability to perform a wide array of natural language tasks. Text-based content moderation is one LLM use case that has received recent enthusiasm, however, there is little research investigating how LLMs perform in content moderation settings. In this work, we evaluate a suite of commodity LLMs on two common content moderation tasks: rule-based community moderation and toxic content detection. For rule-based community moderation, we instantiate 95 subcommunity specific LLMs by prompting GPT-3.5 with rules from 95 Reddit subcommunities. We find that GPT-3.5 is effective at rule-based moderation for many communities, achieving a median accuracy of 64% and a median precision of 83%. For toxicity detection, we evaluate a suite of commodity LLMs (GPT-3, GPT-3.5, GPT-4, Gemini Pro, LLAMA 2) and show that LLMs significantly outperform currently widespread toxicity classifiers. However, recent increases in model size add only marginal benefit to toxicity detection, suggesting a potential performance plateau for LLMs on toxicity detection tasks. We conclude by outlining avenues for future work in studying LLMs and content moderation.