# Language Detoxification with Attribute-Discriminative Latent Space

arXiv: https://arxiv.org/abs/2210.10329 · HTML: https://mlsanity.com/paper/2210.10329/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 40 of its 61 references that are in the map.
- First posted on arXiv 19 Oct 2022.

## Builds on (most cited first, 30 of 40)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://mlsanity.com/paper/1906.08237/) (2019)
- [ALBERT: A Lite BERT for Self-supervised Learning of Language Representations](https://mlsanity.com/paper/1909.11942/) (2019)
- [The Pile: An 800GB Dataset of Diverse Text for Language Modeling](https://mlsanity.com/paper/2101.00027/) (2020)
- [Fine-Tuning Language Models from Human Preferences](https://mlsanity.com/paper/1909.08593/) (2019)
- [Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism](https://mlsanity.com/paper/1909.08053/) (2019)
- [Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context](https://mlsanity.com/paper/1901.02860/) (2019)
- [Reformer: The Efficient Transformer](https://mlsanity.com/paper/2001.04451/) (2020)
- [Don't Stop Pretraining: Adapt Language Models to Domains and Tasks](https://mlsanity.com/paper/2004.10964/) (2020)
- [RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models](https://mlsanity.com/paper/2009.11462/) (2020)
- [Hierarchical Neural Story Generation](https://mlsanity.com/paper/1805.04833/) (2018)
- [A Diversity-Promoting Objective Function for Neural Conversation Models](https://mlsanity.com/paper/1510.03055/) (2015)
- [DialoGPT: Large-Scale Generative Pre-training for Conversational Response Generation](https://mlsanity.com/paper/1911.00536/) (2019)
- [Plug and Play Language Models: A Simple Approach to Controlled Text Generation](https://mlsanity.com/paper/1912.02164/) (2019)
- [CTRL: A Conditional Transformer Language Model for Controllable Generation](https://mlsanity.com/paper/1909.05858/) (2019)
- [Universal Adversarial Triggers for Attacking and Analyzing NLP](https://mlsanity.com/paper/1908.07125/) (2019)
- [The Woman Worked as a Babysitter: On Biases in Language Generation](https://mlsanity.com/paper/1909.01326/) (2019)
- [DExperts: Decoding-Time Controlled Text Generation with Experts and Anti-Experts](https://mlsanity.com/paper/2105.03023/) (2021)
- [GeDi: Generative Discriminator Guided Sequence Generation](https://mlsanity.com/paper/2009.06367/) (2020)
- [SeqGAN: Sequence Generative Adversarial Nets with Policy Gradient](https://mlsanity.com/paper/1609.05473/) (2016)
- [FUDGE: Controlled Text Generation With Future Discriminators](https://mlsanity.com/paper/2104.05218/) (2021)
- [Toward Controlled Generation of Text](https://mlsanity.com/paper/1703.00955/) (2017)
- [Predicting the Type and Target of Offensive Posts in Social Media](https://mlsanity.com/paper/1902.09666/) (2019)
- [A Distributional Approach to Controlled Text Generation](https://mlsanity.com/paper/2012.11635/) (2020)
- [A Knowledge-Enhanced Pretraining Model for Commonsense Story Generation](https://mlsanity.com/paper/2001.05139/) (2020)
- [Optimus: Organizing Sentences via Pre-trained Modeling of a Latent Space](https://mlsanity.com/paper/2004.04092/) (2020)

## Built on by (most cited first, 11 of 11)

- [Self-Detoxifying Language Models via Toxification Reversal](https://mlsanity.com/paper/2310.09573/) (2023)
- [Walking in Others' Shoes: How Perspective-Taking Guides Large Language Models in Reducing Toxicity and Bias](https://mlsanity.com/paper/2407.15366/) (2024)
- [Negating Negatives: Alignment with Human Negative Samples via Distributional Dispreference Optimization](https://mlsanity.com/paper/2403.03419/) (2024)
- [DESTEIN: Navigating Detoxification of Language Models via Universal Steering Pairs and Head-wise Activation Fusion](https://mlsanity.com/paper/2404.10464/) (2024)
- [Learn What NOT to Learn: Towards Generative Safety in Chatbots](https://mlsanity.com/paper/2304.11220/) (2023)
- [UniDetox: Universal Detoxification of Large Language Models via Dataset Distillation](https://mlsanity.com/paper/2504.20500/) (2025)
- [Large Language Models can be Strong Self-Detoxifiers](https://mlsanity.com/paper/2410.03818/) (2024)
- [IF-GUIDE: Influence Function-Guided Detoxification of LLMs](https://mlsanity.com/paper/2506.01790/) (2025)
- [Language Surgery in Multilingual Large Language Models](https://mlsanity.com/paper/2506.12450/) (2025)
- [Test-Time Detoxification without Training or Learning Anything](https://mlsanity.com/paper/2602.02498/) (2026)
- [Fine-Grained Detoxification via Instance-Level Prefixes for Large Language Models](https://mlsanity.com/paper/2402.15202/) (2024)

## Abstract (arXiv)

Transformer-based Language Models (LMs) have achieved impressive results on natural language understanding tasks, but they can also generate toxic text such as insults, threats, and profanity, limiting their real-world applications. To overcome this issue, a few text generation approaches aim to detoxify toxic texts using additional LMs or perturbations. However, previous methods require excessive memory, computations, and time which are serious bottlenecks in their real-world application. To address such limitations, we propose an effective yet efficient method for language detoxification using an attribute-discriminative latent space. Specifically, we project the latent space of an original Transformer LM onto a discriminative latent space that well-separates texts by their attributes using a projection block and an attribute discriminator. This allows the LM to control the text generation to be non-toxic with minimal memory and computation overhead. We validate our model, Attribute-Discriminative Language Model (ADLM) on detoxified language and dialogue generation tasks, on which our method significantly outperforms baselines both in performance and efficiency.