mlsanityData to 7 Oct 2026

A Holistic Approach to Undesired Content Detection in the Real World

arXiv 2208.03274 ↗First posted 5 Aug 2022cs.CL cs.LGcited by 337

Not on the map. Why?

Cited by 160 in LLM safety and security, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We present a holistic approach to building a robust and useful natural language classification system for real-world content moderation. The success of such a system relies on a chain of carefully designed and executed steps, including the design of content taxonomies and labeling instructions, data quality control, an active learning pipeline to capture rare events, and a variety of methods to make the model robust and to avoid overfitting. Our moderation system is trained to detect a broad set of categories of undesired content, including sexual content, hateful content, violence, self-harm, and harassment. This approach generalizes to a wide range of different content taxonomies and can be used to create high-quality content classifiers that outperform off-the-shelf models.

Key facts. Cited by 337 papers in the map (data through 7 Oct 2026). Builds on 54 of its 107 references that are in the map. First posted on arXiv 5 Aug 2022.

Builds on

54 of 107 references.

References · 54

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2014Explaining and Harnessing Adversarial Examplescited by 7,287
  4. 2013Intriguing properties of neural networkscited by 5,114
  5. 2015Domain-Adversarial Training of Neural Networkscited by 3,023
  6. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,168
  7. 2014Unsupervised Domain Adaptation by Backpropagationcited by 1,906
  8. 2017Adversarial Discriminative Domain Adaptationcited by 1,309
  9. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,235
  10. 2022LaMDA: Language Models for Dialog Applicationscited by 1,218
  11. 2022Red Teaming Language Models with Language Modelscited by 983
  12. 2020Beyond Accuracy: Behavioral Testing of NLP models with CheckListcited by 979
38 more
  1. 2021Ethical and social risks of harm from Language Modelscited by 957
  2. 2017Deep Bayesian Active Learning with Image Datacited by 812
  3. 2017Counterfactual Fairnesscited by 802
  4. 2019EDA: Easy Data Augmentation Techniques for Boosting Performance on Text Classification Taskscited by 800
  5. 2017Automated Hate Speech Detection and the Problem of Offensive Languagecited by 724
  6. 2021Dynabench: Rethinking Benchmarking in NLPcited by 447
  7. 2018Generating Natural Language Adversarial Examplescited by 422
  8. 2017Men Also Like Shopping: Reducing Gender Bias Amplification using Corpus-level Constraintscited by 420
  9. 2022ToxiGen: A Large-Scale Machine-Generated Dataset for Adversarial and Implicit Hate Speech Detectioncited by 419
  10. 2020TweetEval: Unified Benchmark and Comparative Evaluation for Tweet Classificationcited by 414
  11. 2019Nuanced Metrics for Measuring Unintended Bias with Real Data for Text Classificationcited by 396
  12. 2021Dealing with Disagreements: Looking Beyond the Majority Vote in Subjective Annotationscited by 306
  13. 2019Predicting the Type and Target of Offensive Posts in Social Mediacited by 268
  14. 2020Learning from the Worst: Dynamically Generated Datasets to Improve Online Hate Detectioncited by 217
  15. 2020HateCheck: Functional Tests for Hate Speech Detection Modelscited by 215
  16. 2019Racial Bias in Hate Speech and Abusive Language Detection Datasetscited by 208
  17. 2018Contextual Augmentation: Data Augmentation by Words with Paradigmatic Relationscited by 201
  18. 2018Hate Speech Dataset from a White Supremacy Forumcited by 196
  19. 2021Want To Reduce Labeling Cost? GPT-3 Can Helpcited by 196
  20. 2021Generating Datasets with Pretrained Language Modelscited by 192
  21. 2021GPT3Mix: Leveraging Large-scale Language Models for Text Augmentationcited by 185
  22. 2022A New Generation of Perspective API: Efficient Multilingual Character-level Transformerscited by 181
  23. 2019Build it Break it Fix it for Dialogue Safety: Robustness from Adversarial Human Attackcited by 176
  24. 2019Not Enough Data? Deep Learning to the Rescue!cited by 174
  25. 2021Challenges in Detoxifying Language Modelscited by 169
  26. 2020Data Augmentation using Pre-trained Transformer Modelscited by 168
  27. 2018Counterfactual Fairness in Text Classification through Robustnesscited by 150
  28. 2020Neural Unsupervised Domain Adaptation in NLP---A Surveycited by 142
  29. 2018Pathologies of Neural Models Make Interpretations Difficultcited by 140
  30. 2020Directions in Abusive Language Training Data: Garbage In, Garbage Outcited by 139
  31. 2019Augmenting Data with Mixup for Sentence Classification: An Empirical Studycited by 125
  32. 2021On Releasing Annotator-Level Labels and Information in Datasetscited by 123
  33. 2020Toxicity Detection: Does Context Really Matter?cited by 93
  34. 2018Deep Bayesian Active Learning for Natural Language Processing: Results of a Large-Scale Empirical Studycited by 90
  35. 2021Towards Zero-Label Language Learningcited by 76
  36. 2020A Simple but Tough-to-Beat Data Augmentation Approach for Natural Language Understanding and Generationcited by 72
  37. 2021Towards generalisable hate speech detection: a review on obstacles and solutionscited by 66
  38. 2022Adversarial Training for High-Stakes Reliabilitycited by 60

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 4; 2023: 38; 2024: 91; 2025: 99; 2026: 105.

337 papers.

Sort:

By area:

  1. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,330
  2. 2024OpenAI o1 System Cardcited by 1,220
  3. 2023Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversationscited by 1,124
  4. 2022Ignore Previous Prompt: Attack Techniques For Language Modelscited by 760
  5. 2023GPTFUZZER: Red Teaming Large Language Models with Auto-Generated Jailbreak Promptscited by 520
  6. 2024WildGuard: Open One-Stop Moderation Tools for Safety Risks, Jailbreaks, and Refusals of LLMscited by 460
  7. 2023"Do Anything Now": Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Modelscited by 457
  8. 2023LMSYS-Chat-1M: A Large-Scale Real-World LLM Conversation Datasetcited by 449
  9. 2023FigStep: Jailbreaking Large Vision-Language Models via Typographic Visual Promptscited by 368
  10. 2023Multi-step Jailbreaking Privacy Attacks on ChatGPTcited by 320
  11. 2023Exploiting Programmatic Behavior of LLMs: Dual-Use Through Standard Security Attackscited by 268
  12. 2024R-Judge: Benchmarking Safety Risk Awareness for LLM Agentscited by 268
38 more
  1. 2023Jailbreak in pieces: Compositional Adversarial Attacks on Multi-Modal Language Modelscited by 260
  2. 2023A Wolf in Sheep's Clothing: Generalized Nested Jailbreak Prompts can Fool Large Language Models Easilycited by 242
  3. 2024ShieldGemma: Generative AI Content Moderation Based on Gemmacited by 224
  4. 2023Multilingual Jailbreak Challenges in Large Language Modelscited by 222
  5. 2024OR-Bench: An Over-Refusal Benchmark for Large Language Modelscited by 222
  6. 2023Survey of Vulnerabilities in Large Language Models Revealed by Adversarial Attackscited by 189
  7. 2024Security and Privacy Challenges of Large Language Models: A Surveycited by 179
  8. 2023One-Dimensional Adapter to Rule Them All: Concepts, Diffusion Models and Erasing Applicationscited by 164
  9. 2024AEGIS: Online Adaptive AI Content Safety Moderation with Ensemble of LLM Expertscited by 126
  10. 2025Qwen3Guard Technical Reportcited by 125
  11. 2024Attacks, Defenses and Evaluations for LLM Conversation Safety: A Surveycited by 124
  12. 2025Aegis2.0: A Diverse AI Safety Dataset and Risks Taxonomy for Alignment of LLM Guardrailscited by 122
  13. 2024GuardAgent: Safeguard LLM Agents by a Guard Agent via Knowledge-Enabled Reasoningcited by 113
  14. 2024Rule Based Rewards for Language Model Safetycited by 106
  15. 2024The Art of Saying No: Contextual Noncompliance in Language Modelscited by 100
  16. 2025GuardReasoner: Towards Reasoning-based LLM Safeguardscited by 89
  17. 2024JailbreakRadar: Comprehensive Assessment of Jailbreak Attacks Against LLMscited by 87
  18. 2024GradSafe: Detecting Jailbreak Prompts for LLMs via Safety-Critical Gradient Analysiscited by 87
  19. 2023A Survey of GPT-3 Family Large Language Models Including ChatGPT and GPT-4cited by 81
  20. 2024Prover-Verifier Games improve legibility of LLM outputscited by 79
  21. 2023Cognitive Overload: Jailbreaking Large Language Models with Overloaded Logical Thinkingcited by 78
  22. 2023Use of LLMs for Illicit Purposes: Threats, Prevention Measures, and Vulnerabilitiescited by 76
  23. 2025LLMs Encode Harmfulness and Refusal Separatelycited by 75
  24. 2024CodeAttack: Revealing Safety Generalization Challenges of Large Language Models via Code Completioncited by 70
  25. 2024Defending Large Language Models against Jailbreak Attacks via Semantic Smoothingcited by 68
  26. 2024Latent Guard: a Safety Framework for Text-to-image Generationcited by 68
  27. 2024RigorLLM: Resilient Guardrails for Large Language Models against Undesired Contentcited by 65
  28. 2024GuardT2I: Defending Text-to-Image Models from Adversarial Promptscited by 61
  29. 2024Refusal-Trained LLMs Are Easily Jailbroken As Browser Agentscited by 61
  30. 2025PolyGuard: A Multilingual Safety Moderation Tool for 17 Languagescited by 57
  31. 2023Self-Guard: Empower the LLM to Safeguard Itselfcited by 52
  32. 2023Privacy in Large Language Models: Attacks, Defenses and Future Directionscited by 48
  33. 2024ShieldLM: Empowering LLMs as Aligned, Customizable and Explainable Safety Detectorscited by 48
  34. 2024Eraser: Jailbreaking Defense in Large Language Models via Unlearning Harmful Knowledgecited by 48
  35. 2023LLM Censorship: A Machine Learning Challenge or a Computer Security Problem?cited by 47
  36. 2023Conversation Chronicles: Towards Diverse Temporal and Relational Dynamics in Multi-Session Conversationscited by 47
  37. 2024SCANS: Mitigating the Exaggerated Safety for LLMs via Safety-Conscious Activation Steeringcited by 45
  38. 2023Evaluating GPT-3 Generated Explanations for Hateful Content Moderationcited by 44

50 most cited of 337 All 337 (JSON)