# Longitudinal Monitoring of LLM Content Moderation of Social Issues

arXiv: https://arxiv.org/abs/2510.01255 · HTML: https://mlsanity.com/paper/2510.01255/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 95 references that are in the map.
- First posted on arXiv 24 Sep 2025.

## Builds on (most cited first, 13 of 13)

- [HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal](https://mlsanity.com/paper/2402.04249/) (2024)
- [Model Cards for Model Reporting](https://mlsanity.com/paper/1810.03993/) (2018)
- [WildGuard: Open One-Stop Moderation Tools for Safety Risks, Jailbreaks, and Refusals of LLMs](https://mlsanity.com/paper/2406.18495/) (2024)
- [Nuanced Metrics for Measuring Unintended Bias with Real Data for Text Classification](https://mlsanity.com/paper/1903.04561/) (2019)
- [A Holistic Approach to Undesired Content Detection in the Real World](https://mlsanity.com/paper/2208.03274/) (2022)
- [AI Transparency in the Age of LLMs: A Human-Centered Research Roadmap](https://mlsanity.com/paper/2306.01941/) (2023)
- [Know Your Limits: A Survey of Abstention in Large Language Models](https://mlsanity.com/paper/2407.18418/) (2024)
- [The Fallacy of AI Functionality](https://mlsanity.com/paper/2206.09511/) (2022)
- [From Hard Refusals to Safe-Completions: Toward Output-Centric Safety Training](https://mlsanity.com/paper/2508.09224/) (2025)
- [Policy-as-Prompt: Rethinking Content Moderation in the Age of Large Language Models](https://mlsanity.com/paper/2502.18695/) (2025)
- [OpenAI's Approach to External Red Teaming for AI Models and Systems](https://mlsanity.com/paper/2503.16431/) (2025)
- [Lost in Moderation: How Commercial Content Moderation APIs Over- and Under-Moderate Group-Targeted Hate Speech and Linguistic Variations](https://mlsanity.com/paper/2503.01623/) (2025)
- [Identity-related Speech Suppression in Generative AI Content Moderation](https://mlsanity.com/paper/2409.13725/) (2024)

## Built on by (most cited first, 2 of 2)

- [Refusal Steering: Fine-grained Control over LLM Refusal Behaviour for Sensitive Topics](https://mlsanity.com/paper/2512.16602/) (2025)
- [Operational Evidence Gaps for LLMs in Fraud Detection and Trust-and-Safety Workflows](https://mlsanity.com/paper/2607.13078/) (2026)

## Abstract (arXiv)

Large language models' (LLMs') outputs are shaped by opaque and frequently-changing company content moderation policies and practices. LLM moderation often takes the form of refusal; models' refusal to produce text about certain topics both reflects company policy and subtly shapes public discourse. We introduce AI Watchman, a longitudinal auditing system to publicly measure and track LLM refusals over time, to provide transparency into an important and black-box aspect of LLMs. Using a dataset of over 400 social issues, we audit Open AI's moderation endpoint, GPT-4.1, and GPT-5, and DeepSeek (both in English and Chinese). We find evidence that changes in company policies, even those not publicly announced, can be detected by AI Watchman, and identify company- and model-specific differences in content moderation. We also qualitatively analyze and categorize different forms of refusal. This work contributes evidence for the value of longitudinal auditing of LLMs, and AI Watchman, one system for doing so.