mlsanityData to 7 Oct 2026

Detoxifying Language Models Risks Marginalizing Minority Voices

arXiv 2104.06390 ↗First posted 13 Apr 2021cs.CL cs.LGcited by 106

Not on the map. Why?

Abstract (arXiv)

Language models (LMs) must be both safe and equitable to be responsibly deployed in practice. With safety in mind, numerous detoxification techniques (e.g., Dathathri et al. 2020; Krause et al. 2020) have been proposed to mitigate toxic LM generations. In this work, we show that current detoxification techniques hurt equity: they decrease the utility of LMs on language used by marginalized groups (e.g., African-American English and minority identity mentions). In particular, we perform automatic and human evaluations of text generation quality when LMs are conditioned on inputs with different dialects and group identifiers. We find that detoxification makes LMs more brittle to distribution shift, especially on language used by marginalized groups. We identify that these failures stem from detoxification methods exploiting spurious correlations in toxicity datasets. Overall, our results highlight the tension between the controllability and distributional robustness of LMs.

Key facts. Cited by 106 papers in the map (data through 7 Oct 2026). Builds on 19 of its 29 references that are in the map. First posted on arXiv 13 Apr 2021.

Builds on

19 of 29 references.

References · 19

  1. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,548
  2. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,235
  3. 2018Hierarchical Neural Story Generationcited by 1,182
  4. 2020Language (Technology) is Power: A Critical Survey of "Bias" in NLPcited by 951
  5. 2019Plug and Play Language Models: A Simple Approach to Controlled Text Generationcited by 793
  6. 2019CTRL: A Conditional Transformer Language Model for Controllable Generationcited by 754
  7. 2019Universal Adversarial Triggers for Attacking and Analyzing NLPcited by 709
  8. 2020Recipes for building an open-domain chatbotcited by 593
  9. 2019Social Bias Frames: Reasoning about Social and Power Implications of Languagecited by 377
  10. 2020GeDi: Generative Discriminator Guided Sequence Generationcited by 377
  11. 2019Don't Take the Easy Way Out: Ensemble Based Methods for Avoiding Known Dataset Biasescited by 279
  12. 2018Examining Gender and Race Bias in Two Hundred Sentiment Analysis Systemscited by 234
7 more
  1. 2020Recipes for Safety in Open-domain Chatbotscited by 168
  2. 2019Distributionally Robust Language Modelingcited by 142
  3. 2020Towards Controllable Biases in Language Generationcited by 112
  4. 2021Challenges in Automated Debiasing for Toxic Language Detectioncited by 95
  5. 2017Ethical Challenges in Data-Driven Dialogue Systemscited by 86
  6. 2020Measuring a hate speech spectrum with faceted Rasch item response theory and perspective-aware, explainable-by-design deep learningcited by 67
  7. 2020Investigating African-American Vernacular English in Transformer-Based Text Generationcited by 52

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 5; 2022: 22; 2023: 35; 2024: 24; 2025: 11; 2026: 9.

106 papers.

Sort:

By area:

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  3. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  4. 2023PaLM 2 Technical Reportcited by 1,154
  5. 2022Red Teaming Language Models with Language Modelscited by 983
  6. 2021Ethical and social risks of harm from Language Modelscited by 957
  7. 2023BloombergGPT: A Large Language Model for Financecited by 923
  8. 2023Bias and Fairness in Large Language Models: A Surveycited by 626
  9. 2022Improving alignment of dialogue agents via targeted human judgementscited by 527
  10. 2022ToxiGen: A Large-Scale Machine-Generated Dataset for Adversarial and Implicit Hate Speech Detectioncited by 419
  11. 2023Visual Adversarial Examples Jailbreak Aligned Large Language Modelscited by 307
  12. 2023Pretraining Language Models with Human Preferencescited by 245
38 more
  1. 2023Automatically Auditing Large Language Models via Discrete Optimizationcited by 187
  2. 2023The political ideology of conversational AI: Converging evidence on ChatGPT's pro-environmental, left-libertarian orientationcited by 183
  3. 2023A Pretrainer's Guide to Training Data: Measuring the Effects of Data Age, Domain Coverage, Quality, & Toxicitycited by 180
  4. 2021Process for Adapting Language Models to Society (PALMS) with Values-Targeted Datasetscited by 179
  5. 2021Challenges in Detoxifying Language Modelscited by 169
  6. 2024Know Your Limits: A Survey of Abstention in Large Language Modelscited by 116
  7. 2023Mechanistically analyzing the effects of fine-tuning on procedurally defined taskscited by 110
  8. 2021Time Waits for No One! Analysis and Challenges of Temporal Misalignmentcited by 91
  9. 2023NLPositionality: Characterizing Design Biases of Datasets and Modelscited by 91
  10. 2023Evaluating the Social Impact of Generative AI Systems in Systems and Societycited by 91
  11. 2022Language Generation Models Can Cause Harm: So What Can We Do About It? An Actionable Surveycited by 80
  12. 2021Anticipating Safety Issues in E2E Conversational AI: Framework and Toolingcited by 77
  13. 2023The Data Provenance Initiative: A Large Scale Audit of Dataset Licensing & Attribution in AIcited by 73
  14. 2022Exploring the Limits of Domain-Adaptive Training for Detoxifying Large-Scale Language Modelscited by 67
  15. 2022Mix and Match: Learning-free Controllable Text Generation using Energy Language Modelscited by 66
  16. 2022Characteristics of Harmful Text: Towards Rigorous Benchmarking of Language Modelscited by 49
  17. 2023Privacy in Large Language Models: Attacks, Defenses and Future Directionscited by 48
  18. 2023Self-Detoxifying Language Models via Toxification Reversalcited by 45
  19. 2024Consent in Crisis: The Rapid Decline of the AI Data Commonscited by 45
  20. 2022Mitigating Gender Bias in Distilled Language Models via Counterfactual Role Reversalcited by 40
  21. 2023A Pathway Towards Responsible AI Generated Contentcited by 38
  22. 2022Detoxifying Text with MaRCo: Controllable Revision with Experts and Anti-Expertscited by 36
  23. 2023Beyond the Safeguards: Exploring the Security Risks of ChatGPTcited by 36
  24. 2024Model Editing as a Robust and Denoised variant of DPO: A Case Study on Toxicitycited by 33
  25. 2022Unified Detoxifying and Debiasing in Language Generation via Inference-time Adaptive Optimizationcited by 32
  26. 2024Get more for less: Principled Data Selection for Warming Up Fine-Tuning in LLMscited by 32
  27. 2023PREADD: Prefix-Adaptive Decoding for Controlled Text Generationcited by 29
  28. 2024A Robot Walks into a Bar: Can Language Models Serve as Creativity Support Tools for Comedy? An Evaluation of LLMs' Humour Alignment with Comedianscited by 29
  29. 2023Integration of Large Language Models and Federated Learningcited by 28
  30. 2024Farsight: Fostering Responsible AI Awareness During AI Application Prototypingcited by 28
  31. 2022Toxicity in Multilingual Machine Translation at Scalecited by 26
  32. 2023KoSBi: A Dataset for Mitigating Social Bias Risks Towards Safer Large Language Model Applicationcited by 24
  33. 2023Goodtriever: Adaptive Toxicity Mitigation with Retrieval-augmented Modelscited by 23
  34. 2024The Typing Cure: Experiences with Large Language Model Chatbots for Mental Health Supportcited by 23
  35. 2023Compositional Capabilities of Autoregressive Transformers: A Study on Synthetic, Interpretable Taskscited by 21
  36. 2024Language Models in Dialogue: Conversational Maxims for Human-AI Interactionscited by 21
  37. 2024Assessing Dialect Fairness and Robustness of Large Language Models in Reasoning Taskscited by 20
  38. 2025Rejected Dialects: Biases Against African American Language in Reward Modelscited by 20

50 most cited of 106 All 106 (JSON)