mlsanityData to 14 Oct 2026

Policy-as-Prompt: Rethinking Content Moderation in the Age of Large Language Models

arXiv 2502.18695 ↗First posted 25 Feb 2025cs.CY cs.AI cs.SIcited by 31

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

Content moderation plays a critical role in shaping safe and inclusive online environments, balancing platform standards, user expectations, and regulatory frameworks. Traditionally, this process involves operationalising policies into guidelines, which are then used by downstream human moderators for enforcement, or to further annotate datasets for training machine learning moderation models. However, recent advancements in large language models (LLMs) are transforming this landscape. These models can now interpret policies directly as textual inputs, eliminating the need for extensive data curation. This approach offers unprecedented flexibility, as moderation can be dynamically adjusted through natural language interactions. This paradigm shift raises important questions about how policies are operationalised and the implications for content moderation practices. In this paper, we formalise the emerging policy-as-prompt framework and identify five key challenges across four domains: Technical Implementation (1. translating policy to prompts, 2. sensitivity to prompt structure and formatting), Sociotechnical (3. the risk of technological determinism in policy formation), Organisational (4. evolving roles between policy and machine learning teams), and Governance (5. model governance and accountability). Through analysing these challenges across technical, sociotechnical, organisational, and governance dimensions, we discuss potential mitigation approaches. This research provides actionable insights for practitioners and lays the groundwork for future exploration of scalable and adaptive content moderation systems in digital ecosystems.

Key facts. Cited by 31 papers in the map (data through 14 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 16 of its 71 references that are in the map. First posted on arXiv 25 Feb 2025.

Builds on

16 of 71 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024Do LLMs Exhibit Human-Like Reasoning? Evaluating Theory of Mind in LLMs for Open-Ended Responses9 Jun
  2. 2024Rethinking Interpretability in the Era of Large Language Models30 Jan

Other references · 14

  1. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  2. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,375
  3. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
  4. 2023Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversationscited by 1,207
  5. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learnedcited by 729
  6. 2024A Systematic Survey of Prompt Engineering in Large Language Models: Techniques and Applicationscited by 431
  7. 2024Does Prompt Formatting Have Any Impact on LLM Performance?cited by 141
  8. 2024The Prompt Report: A Systematic Survey of Prompt Engineering Techniquescited by 121
  9. 2024Meta-Prompting: Enhancing Language Models with Task-Agnostic Scaffoldingcited by 117
  10. 2024CMAT: A Multi-Agent Collaboration Tuning Framework for Enhancing Small Language Modelscited by 26
  11. 2024Advancing Content Moderation: Evaluating Large Language Models for Detecting Sensitive Content Across Text, Images, and Videoscited by 10
  12. 2024Concentrate Attention: Towards Domain-Generalizable Prompt Optimization for Language Modelscited by 7
2 more
  1. 2024Attention Instruction: Amplifying Attention in the Middle via Promptingcited by 6
  2. 2024Gen-AI for User Safety: A Surveycited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 14; 2026: 17.

31 papers.

Sort:

By area:

  1. 2026Institutional AI: Governing LLM Collusion in Multi-Agent Cournot Markets via Public Governance Graphscited by 20
  2. 2025MoMoE: Mixture of Moderation Experts Framework for AI-Assisted Online Governancecited by 13
  3. 2025LLM Prompt Duel Optimizer: Efficient Label-Free Prompt Optimizationcited by 8
  4. 2026The Impact of AI-Generated Text on the Internetcited by 6
  5. 2025Evaluating the Robustness of Large Language Model Safety Guardrails Against Adversarial Attackscited by 5
  6. 2025Sword and Shield: Uses and Strategies of LLMs in Navigating Disinformationcited by 4
  7. 2025UWBench: A Comprehensive Vision-Language Benchmark for Underwater Understandingcited by 4
  8. 2025Dynamic Content Moderation in Livestreams: Combining Supervised Classification with MLLM-Boosted Similarity Matchingcited by 4
  9. 2025Evaluating Large Language Models for Detecting Antisemitismcited by 3
  10. 2025AI Security Beyond Core Domains: Resume Screening as a Case Study of Adversarial Vulnerabilities in Specialized LLM Applicationscited by 3
  11. 2025Longitudinal Monitoring of LLM Content Moderation of Social Issuescited by 2
  12. 2026Position: Capability Control Should be a Separate Goal From Alignmentcited by 2
19 more
  1. 2026The Consensus Trap: Dissecting Subjectivity and the "Ground Truth" Illusion in Data Annotationcited by 2
  2. 2025The Problem with Safety Classification is not just the Modelscited by 1
  3. 2025Evaluating LLMs for Demographic-Targeted Social Bias Detection: A Comprehensive Benchmark Studycited by 1
  4. 2025Ideology-Based LLMs for Content Moderationcited by 1
  5. 2026Solipsistic Superintelligence is Unlikely to be Cooperativecited by 1
  6. 2026MatchLM2Lite: A Scalable MLLM-to-Lite Framework for Reproduced Content Identificationcited by 1
  7. 2026From Failure Taxonomy to Intervention: A Diagnostic Methodology for Industry-Scale AVLM in Video and Live-Streaming Platform Moderationcited by 1
  8. 2026Can Foundation Models Moderate Online Content? Evaluating Instruction- vs. Example-Driven Policy Operationalizationcited by 1
  9. 2025Quantifying Feature Importance for Online Content Moderationcited by 0
  10. 2025Malicious Image Analysis via Vision-Language Segmentation Fusion: Detection, Element, and Location in One-shotcited by 0
  11. 2026GMP: A Benchmark for Content Moderation under Co-occurring Violations and Dynamic Rulescited by 0
  12. 2026CHAIRO: Contextual Hierarchical Analogical Induction and Reasoning Optimization for LLMscited by 0
  13. 2026CARO: Chain-of-Analogy Reasoning Optimization for Robust Content Moderationcited by 0
  14. 2026Operational Evidence Gaps for LLMs in Fraud Detection and Trust-and-Safety Workflowscited by 0
  15. 2026Semi-Automated Detection of Gaps in LLM Security Knowledgecited by 0
  16. 2026Whitewashing Hate, Smearing Harmless Content: Annotator-Style Rebuttal Attacks on LLM-Based Moderationcited by 0
  17. 2026Evaluating Criterion-Conditioned Behaviour of Large Language Models in Content Moderationcited by 0
  18. 2026Alignment Inertia: Auditing the Durability of Training Data Influence Through Policy Override Resistancecited by 0
  19. 2026AdaptEvo: Adaptive Agent Learning with Evolving Supervisioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Open-Domain Safety Policy Construction
  2. 2025Scaling Reinforcement Learning for Content Moderation with Large Language Models
  3. 2025PAM: Training Policy-Aligned Moderation Filters at Scale
  4. 2025Towards Trustworthy Multimodal Moderation via Policy-Aligned Reasoning and Hierarchical Labeling
  5. 2026"AI Watermarking": Bridging Policy Discourse and Technical Capabilities
  6. 2025Prompt Commons: Collective Prompting as Governance for Urban AI
  7. 2026Characterizing Bluesky Content Moderation Service: From Automation of Service to Landscape of Harms
  8. 2026PolicyKG: An Agentic LLM Pipeline for Translating Institutional Policies into SHACL Knowledge Graphs