mlsanityData to 7 Oct 2026

Language Detoxification with Attribute-Discriminative Latent Space

arXiv 2210.10329 ↗First posted 19 Oct 2022cs.CL cs.AIcited by 11

Not on the map. Why?

Abstract (arXiv)

Transformer-based Language Models (LMs) have achieved impressive results on natural language understanding tasks, but they can also generate toxic text such as insults, threats, and profanity, limiting their real-world applications. To overcome this issue, a few text generation approaches aim to detoxify toxic texts using additional LMs or perturbations. However, previous methods require excessive memory, computations, and time which are serious bottlenecks in their real-world application. To address such limitations, we propose an effective yet efficient method for language detoxification using an attribute-discriminative latent space. Specifically, we project the latent space of an original Transformer LM onto a discriminative latent space that well-separates texts by their attributes using a projection block and an attribute discriminator. This allows the LM to control the text generation to be non-toxic with minimal memory and computation overhead. We validate our model, Attribute-Discriminative Language Model (ADLM) on detoxified language and dialogue generation tasks, on which our method significantly outperforms baselines both in performance and efficiency.

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Builds on 40 of its 61 references that are in the map. First posted on arXiv 19 Oct 2022.

Builds on

40 of 61 references.

References · 40

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  5. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  6. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  7. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  8. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  9. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  10. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  11. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  12. 2020Reformer: The Efficient Transformercited by 1,658
28 more
  1. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,548
  2. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,235
  3. 2018Hierarchical Neural Story Generationcited by 1,182
  4. 2015A Diversity-Promoting Objective Function for Neural Conversation Modelscited by 1,068
  5. 2019DialoGPT: Large-Scale Generative Pre-training for Conversational Response Generationcited by 842
  6. 2019Plug and Play Language Models: A Simple Approach to Controlled Text Generationcited by 793
  7. 2019CTRL: A Conditional Transformer Language Model for Controllable Generationcited by 754
  8. 2019Universal Adversarial Triggers for Attacking and Analyzing NLPcited by 709
  9. 2019The Woman Worked as a Babysitter: On Biases in Language Generationcited by 540
  10. 2021DExperts: Decoding-Time Controlled Text Generation with Experts and Anti-Expertscited by 434
  11. 2020GeDi: Generative Discriminator Guided Sequence Generationcited by 377
  12. 2016SeqGAN: Sequence Generative Adversarial Nets with Policy Gradientcited by 375
  13. 2021FUDGE: Controlled Text Generation With Future Discriminatorscited by 374
  14. 2017Toward Controlled Generation of Textcited by 271
  15. 2019Predicting the Type and Target of Offensive Posts in Social Mediacited by 268
  16. 2020A Distributional Approach to Controlled Text Generationcited by 117
  17. 2020A Knowledge-Enhanced Pretraining Model for Commonsense Story Generationcited by 116
  18. 2020Optimus: Organizing Sentences via Pre-trained Modeling of a Latent Spacecited by 114
  19. 2021Detoxifying Language Models Risks Marginalizing Minority Voicescited by 106
  20. 2021Challenges in Automated Debiasing for Toxic Language Detectioncited by 95
  21. 2021What's in the Box? A Preliminary Analysis of Undesirable Content in the Common Crawl Corpuscited by 79
  22. 2021On the Safety of Conversational Models: Taxonomy, Dataset, and Benchmarkcited by 73
  23. 2020Contrastive Learning with Adversarial Perturbations for Conditional Text Generationcited by 56
  24. 2021Just Say No: Analyzing the Stance of Neural Dialogue Generation in Offensive Contextscited by 56
  25. 2020Investigating African-American Vernacular English in Transformer-Based Text Generationcited by 52
  26. 2021Text Detoxification using Large Pre-trained Neural Modelscited by 52
  27. 2020CoCon: A Self-Supervised Approach for Controlled Text Generationcited by 47
  28. 2021Modulating Language Models with Emotionscited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 5; 2025: 3; 2026: 1.

11 papers.

  1. 2023Self-Detoxifying Language Models via Toxification Reversalcited by 45
  2. 2024Walking in Others' Shoes: How Perspective-Taking Guides Large Language Models in Reducing Toxicity and Biascited by 17
  3. 2024Negating Negatives: Alignment with Human Negative Samples via Distributional Dispreference Optimizationcited by 11
  4. 2024DESTEIN: Navigating Detoxification of Language Models via Universal Steering Pairs and Head-wise Activation Fusioncited by 9
  5. 2025UniDetox: Universal Detoxification of Large Language Models via Dataset Distillationcited by 8
  6. 2023Learn What NOT to Learn: Towards Generative Safety in Chatbotscited by 7
  7. 2024Large Language Models can be Strong Self-Detoxifierscited by 7
  8. 2025IF-GUIDE: Influence Function-Guided Detoxification of LLMscited by 7
  9. 2025Language Surgery in Multilingual Large Language Modelscited by 4
  10. 2026Test-Time Detoxification without Training or Learning Anythingcited by 2
  11. 2024Fine-Grained Detoxification via Instance-Level Prefixes for Large Language Modelscited by 0