# Content Moderation by LLM: From Accuracy to Legitimacy

arXiv: https://arxiv.org/abs/2409.03219 · HTML: https://mlsanity.com/paper/2409.03219/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- Builds on 49 of its 174 references that are in the map.
- First posted on arXiv 5 Sep 2024.

## Builds on (most cited first, 30 of 49)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions](https://mlsanity.com/paper/2311.05232/) (2023)
- [Towards Understanding Sycophancy in Language Models](https://mlsanity.com/paper/2310.13548/) (2023)
- [Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMs](https://mlsanity.com/paper/2306.13063/) (2023)
- [Ethical and social risks of harm from Language Models](https://mlsanity.com/paper/2112.04359/) (2021)
- [ChatGPT Outperforms Crowd-Workers for Text-Annotation Tasks](https://mlsanity.com/paper/2303.15056/) (2023)
- [Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Models](https://mlsanity.com/paper/2309.01219/) (2023)
- [Bias and Fairness in Large Language Models: A Survey](https://mlsanity.com/paper/2309.00770/) (2023)
- [HateXplain: A Benchmark Dataset for Explainable Hate Speech Detection](https://mlsanity.com/paper/2012.10289/) (2020)
- [Hallucination is Inevitable: An Innate Limitation of Large Language Models](https://mlsanity.com/paper/2401.11817/) (2024)
- [Simple synthetic data reduces sycophancy in large language models](https://mlsanity.com/paper/2308.03958/) (2023)
- [LLM Lies: Hallucinations are not Bugs, but Features as Adversarial Examples](https://mlsanity.com/paper/2310.01469/) (2023)
- [Is ChatGPT better than Human Annotators? Potential and Limitations of ChatGPT in Explaining Implicit Hate Speech](https://mlsanity.com/paper/2302.07736/) (2023)
- [Quantifying Uncertainty in Answers from any Language Model and Enhancing their Trustworthiness](https://mlsanity.com/paper/2308.16175/) (2023)
- [Rethinking Interpretability in the Era of Large Language Models](https://mlsanity.com/paper/2402.01761/) (2024)
- [Large Language Models are Geographically Biased](https://mlsanity.com/paper/2402.02680/) (2024)
- [The Dawn After the Dark: An Empirical Study on Factuality Hallucination in Large Language Models](https://mlsanity.com/paper/2401.03205/) (2024)
- [Whose Ground Truth? Accounting for Individual and Collective Identities Underlying Dataset Annotation](https://mlsanity.com/paper/2112.04554/) (2021)
- [Probing LLMs for hate speech detection: strengths and vulnerabilities](https://mlsanity.com/paper/2310.12860/) (2023)
- [Can LLM Graph Reasoning Generalize beyond Pattern Memorization?](https://mlsanity.com/paper/2406.15992/) (2024)
- [Evaluating GPT-3 Generated Explanations for Hateful Content Moderation](https://mlsanity.com/paper/2305.17680/) (2023)
- ["HOT" ChatGPT: The promise of ChatGPT in detecting and discriminating hateful, offensive, and toxic comments on social media](https://mlsanity.com/paper/2304.10619/) (2023)
- [Robustifying Safety-Aligned Large Language Models through Clean Data Curation](https://mlsanity.com/paper/2405.19358/) (2024)
- [Can Large Language Models Capture Dissenting Human Voices?](https://mlsanity.com/paper/2305.13788/) (2023)
- [The Perils & Promises of Fact-checking with Large Language Models](https://mlsanity.com/paper/2310.13549/) (2023)
- [Data Augmentation using Large Language Models: Data Perspectives, Learning Paradigms and Challenges](https://mlsanity.com/paper/2403.02990/) (2024)
- [On the Independence of Association Bias and Empirical Fairness in Language Models](https://mlsanity.com/paper/2304.10153/) (2023)
- [Adapting Large Language Models for Content Moderation: Pitfalls in Data Engineering and Supervised Fine-tuning](https://mlsanity.com/paper/2310.03400/) (2023)
- [Properties and Challenges of LLM-Generated Explanations](https://mlsanity.com/paper/2402.10532/) (2024)
- [Automated Claim Matching with Large Language Models: Empowering Fact-Checkers in the Fight Against Misinformation](https://mlsanity.com/paper/2310.09223/) (2023)

## Built on by (most cited first, 30 of 30)

- [MoMoE: Mixture of Moderation Experts Framework for AI-Assisted Online Governance](https://mlsanity.com/paper/2505.14483/) (2025)
- [Hallucination to Truth: A Review of Fact-Checking and Factuality Evaluation in Large Language Models](https://mlsanity.com/paper/2508.03860/) (2025)
- [AI vs. Human Moderators: A Comparative Evaluation of Multimodal LLMs in Content Moderation for Brand Safety](https://mlsanity.com/paper/2508.05527/) (2025)
- [AI and the Future of Digital Public Squares](https://mlsanity.com/paper/2412.09988/) (2024)
- [Class-RAG: Real-Time Content Moderation with Retrieval Augmented Generation](https://mlsanity.com/paper/2410.14881/) (2024)
- [Evaluating LLM-Generated Legal Explanations for Regulatory Compliance in Social Media Influencer Marketing](https://mlsanity.com/paper/2510.08111/) (2025)
- [LLM Performance Predictors: Learning When to Escalate in Hybrid Human-AI Moderation Systems](https://mlsanity.com/paper/2601.07006/) (2026)
- [Evaluating Large Language Models for Detecting Antisemitism](https://mlsanity.com/paper/2509.18293/) (2025)
- [AI Security Beyond Core Domains: Resume Screening as a Case Study of Adversarial Vulnerabilities in Specialized LLM Applications](https://mlsanity.com/paper/2512.20164/) (2025)
- [Revealing Hidden Mechanisms of Cross-Country Content Moderation with Natural Language Processing](https://mlsanity.com/paper/2503.05280/) (2025)
- [Aetheria: A multimodal interpretable content safety framework based on multi-agent debate and collaboration](https://mlsanity.com/paper/2512.02530/) (2025)
- [Black Box Absorption: LLMs Undermining Innovative Ideas](https://mlsanity.com/paper/2510.20612/) (2025)
- [Transforming User Defined Criteria into Explainable Indicators with an Integrated LLM AHP System](https://mlsanity.com/paper/2601.05267/) (2025)
- [From Failure Taxonomy to Intervention: A Diagnostic Methodology for Industry-Scale AVLM in Video and Live-Streaming Platform Moderation](https://mlsanity.com/paper/2606.30059/) (2026)
- [FLAME: Flexible LLM-Assisted Moderation Engine](https://mlsanity.com/paper/2502.09175/) (2025)
- [Are LLMs Good Safety Agents or a Propaganda Engine?](https://mlsanity.com/paper/2511.23174/) (2025)
- [When Harmful Content Gets Camouflaged: Unveiling Perception Failure of LVLMs with CamHarmTI](https://mlsanity.com/paper/2512.03087/) (2025)
- [Improving Implicit Hate Speech Detection via a Community-Driven Multi-Agent Framework](https://mlsanity.com/paper/2601.09342/) (2026)
- [GMP: A Benchmark for Content Moderation under Co-occurring Violations and Dynamic Rules](https://mlsanity.com/paper/2603.01724/) (2026)
- [CHAIRO: Contextual Hierarchical Analogical Induction and Reasoning Optimization for LLMs](https://mlsanity.com/paper/2604.10502/) (2026)
- [Moltbook Moderation: Uncovering Hidden Intent Through Multi-Turn Dialogue](https://mlsanity.com/paper/2605.12856/) (2026)
- [SEP-Attack: A Simple and Effective Paradigm for Transfer-Based Textual Adversarial Attack](https://mlsanity.com/paper/2605.24958/) (2026)
- [Evolving and Detecting Multi-Turn Deception using Geometric Signatures](https://mlsanity.com/paper/2605.27671/) (2026)
- [A Survey of Toxicity Detection and Mitigation Strategies for Multilingual Language Models](https://mlsanity.com/paper/2606.25380/) (2026)
- [Understanding Interpretation Difficulty in Harmful Online Communication: Insights from Cybercrime Communities](https://mlsanity.com/paper/2607.07277/) (2026)
- [Operational Evidence Gaps for LLMs in Fraud Detection and Trust-and-Safety Workflows](https://mlsanity.com/paper/2607.13078/) (2026)
- [Auditing Exposure to Harmful Content on TikTok using Multimodal Language Models: A Cross-National, Age-Stratified Study](https://mlsanity.com/paper/2608.17583/) (2026)
- [From Specialization to Generalization: Instruction-tuned LLMs for Robust Harmful Content Mitigation](https://mlsanity.com/paper/2608.25605/) (2026)
- [Detoxifying Toxic Communication: A Design Science Approach to Responsible AI](https://mlsanity.com/paper/2609.00361/) (2026)
- [Evaluating Criterion-Conditioned Behaviour of Large Language Models in Content Moderation](https://mlsanity.com/paper/2609.03814/) (2026)

## Abstract (arXiv)

One trending application of LLM (large language model) is to use it for content moderation in online platforms. Most current studies on this application have focused on the metric of accuracy -- the extent to which LLMs make correct decisions about content. This article argues that accuracy is insufficient and misleading because it fails to grasp the distinction between easy cases and hard cases, as well as the inevitable trade-offs in achieving higher accuracy. Closer examination reveals that content moderation is a constitutive part of platform governance, the key of which is to gain and enhance legitimacy. Instead of making moderation decisions correct, the chief goal of LLMs is to make them legitimate. In this regard, this article proposes a paradigm shift from the single benchmark of accuracy towards a legitimacy-based framework for evaluating the performance of LLM moderators. The framework suggests that for easy cases, the key is to ensure accuracy, speed, and transparency, while for hard cases, what matters is reasoned justification and user participation. Examined under this framework, LLMs' real potential in moderation is not accuracy improvement. Rather, LLMs can better contribute in four other aspects: to conduct screening of hard cases from easy cases, to provide quality explanations for moderation decisions, to assist human reviewers in getting more contextual information, and to facilitate user participation in a more interactive way. To realize these contributions, this article proposes a workflow for incorporating LLMs into the content moderation system. Using normative theories from law and social sciences to critically assess the new technological application, this article seeks to redefine LLMs' role in content moderation and redirect relevant research in this field.