# The Gray Area: Characterizing Moderator Disagreement on Reddit

arXiv: https://arxiv.org/abs/2601.01620 · HTML: https://mlsanity.com/paper/2601.01620/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 68 references that are in the map.
- First posted on arXiv 4 Jan 2026.

## Builds on (most cited first, 16 of 16)

- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Datasheets for Datasets](https://mlsanity.com/paper/1803.09010/) (2018)
- [BERTopic: Neural topic modeling with a class-based TF-IDF procedure](https://mlsanity.com/paper/2203.05794/) (2022)
- [Understanding Dataset Difficulty with V-Usable Information](https://mlsanity.com/paper/2110.08420/) (2021)
- [A Theory of Usable Information Under Computational Constraints](https://mlsanity.com/paper/2002.10689/) (2020)
- [Toward a Perspectivist Turn in Ground Truthing for Predictive Computing](https://mlsanity.com/paper/2109.04270/) (2021)
- [People Make Better Edits: Measuring the Efficacy of LLM-Generated Counterfactually Augmented Data for Harmful Language Detection](https://mlsanity.com/paper/2311.01270/) (2023)
- [Birdwatch: Crowd Wisdom and Bridging Algorithms can Inform Understanding and Reduce the Spread of Misinformation](https://mlsanity.com/paper/2210.15723/) (2022)
- [Detecting Community Sensitive Norm Violations in Online Conversations](https://mlsanity.com/paper/2110.04419/) (2021)
- [MoMoE: Mixture of Moderation Experts Framework for AI-Assisted Online Governance](https://mlsanity.com/paper/2505.14483/) (2025)
- ["Community Guidelines Make this the Best Party on the Internet": An In-Depth Study of Online Platforms' Content Moderation Policies](https://mlsanity.com/paper/2405.05225/) (2024)
- [Making Online Communities 'Better': A Taxonomy of Community Values on Reddit](https://mlsanity.com/paper/2109.05152/) (2021)
- [CPL-NoViD: Context-Aware Prompt-based Learning for Norm Violation Detection in Online Communities](https://mlsanity.com/paper/2305.09846/) (2023)
- [Venire: A Machine Learning-Guided Panel Review System for Community Content Moderation](https://mlsanity.com/paper/2410.23448/) (2024)
- [AppealMod: Inducing Friction to Reduce Moderator Workload of Handling User Appeals](https://mlsanity.com/paper/2301.07163/) (2023)
- [Characterizing Political Campaigning with Lexical Mutants on Indian Social Media](https://mlsanity.com/paper/2401.03533/) (2024)

## Built on by (most cited first, 3 of 3)

- [Beyond Binary Preferences: A Principled Framework for Reward Modeling with Ordinal Feedback](https://mlsanity.com/paper/2603.02232/) (2026)
- [Can Foundation Models Moderate Online Content? Evaluating Instruction- vs. Example-Driven Policy Operationalization](https://mlsanity.com/paper/2609.10410/) (2026)
- [From Web(logs) to Web(AI): Questions, Platforms, and Methods across Twenty Editions of ICWSM](https://mlsanity.com/paper/2610.00205/) (2026)

## Abstract (arXiv)

Volunteer moderators play a crucial role in sustaining online dialogue, but they often disagree about what should or should not be allowed. In this paper, we study the complexity of content moderation with a focus on disagreements between moderators, which we term the ``gray area'' of moderation. Leveraging 5 years and 4.3 million moderation log entries from 24 subreddits of different topics and sizes, we characterize how gray area, or disputed cases, differ from undisputed cases. We show that one-in-seven moderation cases are disputed among moderators, often addressing transgressions where users' intent is not directly legible, such as in trolling and brigading, as well as tensions around community governance. This is concerning, as almost half of all gray area cases involved automated moderation decisions. Through information-theoretic evaluations, we demonstrate that gray area cases are inherently harder to adjudicate than undisputed cases and show that state-of-the-art language models struggle to adjudicate them. We highlight the key role of expert human moderators in overseeing the moderation process and provide insights about the challenges of current moderation processes and tools.