mlsanityData to 7 Oct 2026

Challenges in Detoxifying Language Models

arXiv 2109.07445 ↗First posted 15 Sep 2021cs.CL cs.AI cs.CYcited by 169

Not on the map. Why?

Cited by 37 in LLM safety and security, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Large language models (LM) generate remarkably fluent text and can be efficiently adapted across NLP tasks. Measuring and guaranteeing the quality of generated text in terms of safety is imperative for deploying LMs in the real world; to this end, prior work often relies on automatic evaluation of LM toxicity. We critically discuss this approach, evaluate several toxicity mitigation strategies with respect to both automatic and human evaluation, and analyze consequences of toxicity mitigation in terms of model bias and LM quality. We demonstrate that while basic intervention strategies can effectively optimize previously established automatic metrics on the RealToxicityPrompts dataset, this comes at the cost of reduced LM coverage for both texts about, and dialects of, marginalized groups. Additionally, we find that human raters often disagree with high automatic toxicity scores after strong toxicity reduction interventions -- highlighting further the nuances involved in careful evaluation of LM toxicity.

Key facts. Cited by 169 papers in the map (data through 7 Oct 2026). Builds on 45 of its 63 references that are in the map. First posted on arXiv 15 Sep 2021.

Builds on

45 of 63 references.

References · 45

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2016Pointer Sentinel Mixture Modelscited by 2,992
  6. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  7. 2020Extracting Training Data from Large Language Modelscited by 2,144
  8. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  9. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  10. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,548
  11. 2019Energy and Policy Considerations for Deep Learning in NLPcited by 1,242
  12. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,235
33 more
  1. 2015A Diversity-Promoting Objective Function for Neural Conversation Modelscited by 1,068
  2. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 960
  3. 2019Plug and Play Language Models: A Simple Approach to Controlled Text Generationcited by 793
  4. 2020The State and Fate of Linguistic Diversity and Inclusion in the NLP Worldcited by 789
  5. 2017Automated Hate Speech Detection and the Problem of Offensive Languagecited by 724
  6. 2019Universal Adversarial Triggers for Attacking and Analyzing NLPcited by 709
  7. 2019The Woman Worked as a Babysitter: On Biases in Language Generationcited by 540
  8. 2021BOLD: Dataset and Metrics for Measuring Biases in Open-Ended Language Generationcited by 410
  9. 2019Nuanced Metrics for Measuring Unintended Bias with Real Data for Text Classificationcited by 396
  10. 2019Social Bias Frames: Reasoning about Social and Power Implications of Languagecited by 377
  11. 2020GeDi: Generative Discriminator Guided Sequence Generationcited by 377
  12. 2021Persistent Anti-Muslim Bias in Large Language Modelscited by 376
  13. 2021Self-Diagnosis and Self-Debiasing: A Proposal for Reducing Corpus-Based Bias in NLPcited by 336
  14. 2015A note on the evaluation of generative modelscited by 291
  15. 2019Identifying and Reducing Gender Bias in Word-Level Language Modelscited by 214
  16. 2016Ex Machina: Personal Attacks Seen at Scalecited by 208
  17. 2019Racial Bias in Hate Speech and Abusive Language Detection Datasetscited by 208
  18. 2019SemEval-2019 Task 6: Identifying and Categorizing Offensive Language in Social Media (OffensEval)cited by 206
  19. 2016Demographic Dialectal Variation in Social Media: A Case Study of African-American Englishcited by 194
  20. 2020Recipes for Safety in Open-domain Chatbotscited by 168
  21. 2018Reducing Gender Bias in Abusive Language Detectioncited by 152
  22. 2019Reducing Sentiment Bias in Language Models via Counterfactual Evaluationcited by 148
  23. 2020A Distributional Approach to Controlled Text Generationcited by 117
  24. 2021Detoxifying Language Models Risks Marginalizing Minority Voicescited by 106
  25. 2020Automatically Identifying Words That Can Serve as Labels for Few-Shot Text Classificationcited by 98
  26. 2017Measuring the Reliability of Hate Speech Annotations: The Case of the European Refugee Crisiscited by 95
  27. 2021Challenges in Automated Debiasing for Toxic Language Detectioncited by 95
  28. 2020"What We Can't Measure, We Can't Understand": Challenges to Demographic Data Procurement in the Pursuit of Fairnesscited by 56
  29. 2018Fighting Offensive Language on Social Media with Unsupervised Text Style Transfercited by 55
  30. 2021Fairness for Unobserved Characteristics: Insights from Technological Impacts on Queer Communitiescited by 40
  31. 2021Civil Rephrases Of Toxic Texts With Self-Supervised Transformerscited by 32
  32. 2020Plug-and-Play Conversational Modelscited by 30
  33. 2018But Who Protects the Moderators? The Case of Crowdsourced Image Moderationcited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 3; 2022: 26; 2023: 61; 2024: 41; 2025: 25; 2026: 13.

169 papers.

Sort:

By area:

  1. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  2. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  3. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,565
  4. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,023
  5. 2022Red Teaming Language Models with Language Modelscited by 983
  6. 2021Ethical and social risks of harm from Language Modelscited by 957
  7. 2023BloombergGPT: A Large Language Model for Financecited by 923
  8. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 890
  9. 2023The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Onlycited by 755
  10. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 743
  11. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learnedcited by 702
  12. 2022Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Modelcited by 557
38 more
  1. 2022Improving alignment of dialogue agents via targeted human judgementscited by 527
  2. 2023GPTFUZZER: Red Teaming Large Language Models with Auto-Generated Jailbreak Promptscited by 520
  3. 2023XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Modelscited by 477
  4. 2023GPT-4 Is Too Smart To Be Safe: Stealthy Chat with LLMs via Ciphercited by 401
  5. 2023Trustworthy LLMs: a Survey and Guideline for Evaluating Large Language Models' Alignmentcited by 394
  6. 2022A Holistic Approach to Undesired Content Detection in the Real Worldcited by 337
  7. 2023Visual Adversarial Examples Jailbreak Aligned Large Language Modelscited by 307
  8. 2023Are aligned neural networks adversarially aligned?cited by 305
  9. 2023Pretraining Language Models with Human Preferencescited by 245
  10. 2023Multilingual Jailbreak Challenges in Large Language Modelscited by 222
  11. 2023Survey of Vulnerabilities in Large Language Models Revealed by Adversarial Attackscited by 189
  12. 2023A Pretrainer's Guide to Training Data: Measuring the Effects of Data Age, Domain Coverage, Quality, & Toxicitycited by 180
  13. 2023A Categorical Archive of ChatGPT Failurescited by 167
  14. 2023Conformal Language Modelingcited by 163
  15. 2023Sociotechnical Safety Evaluation of Generative AI Systemscited by 154
  16. 2023Auditing large language models: a three-layered approachcited by 135
  17. 2024Rho-1: Not All Tokens Are What You Needcited by 132
  18. 2024OpenFedLLM: Training Large Language Models on Decentralized Private Data via Federated Learningcited by 124
  19. 2024AI Agents Under Threat: A Survey of Key Security Challenges and Future Pathwayscited by 122
  20. 2023Aligning Language Models with Preferences through f-divergence Minimizationcited by 118
  21. 2023Mechanistically analyzing the effects of fine-tuning on procedurally defined taskscited by 110
  22. 2023KGA: A General Machine Unlearning Framework Based on Knowledge Gap Alignmentcited by 94
  23. 2023Language Model Behavior: A Comprehensive Surveycited by 92
  24. 2023Identifying and Mitigating the Security Risks of Generative AIcited by 90
  25. 2025GuardReasoner: Towards Reasoning-based LLM Safeguardscited by 89
  26. 2022On Reinforcement Learning and Distribution Matching for Fine-Tuning Language Models with no Catastrophic Forgettingcited by 80
  27. 2022Controllable Natural Language Generation with Contrastive Prefixescited by 75
  28. 2023The Data Provenance Initiative: A Large Scale Audit of Dataset Licensing & Attribution in AIcited by 73
  29. 2024Safeguarding Large Language Models: A Surveycited by 73
  30. 2024FlipAttack: Jailbreak LLMs via Flippingcited by 73
  31. 2023Large Language Models are Not Yet Human-Level Evaluators for Abstractive Summarizationcited by 70
  32. 2024Safety of Multimodal Large Language Models on Images and Textscited by 70
  33. 2024When LLMs Meet Cybersecurity: A Systematic Literature Reviewcited by 70
  34. 2022Exploring the Limits of Domain-Adaptive Training for Detoxifying Large-Scale Language Modelscited by 67
  35. 2023Shall We Pretrain Autoregressive Language Models with Retrieval? A Comprehensive Studycited by 63
  36. 2024Risk Taxonomy, Mitigation, and Assessment Benchmarks of Large Language Model Systemscited by 63
  37. 2022In conversation with Artificial Intelligence: aligning language models with human valuescited by 61
  38. 2024Building Guardrails for Large Language Modelscited by 61

50 most cited of 169 All 169 (JSON)