# Text Detoxification using Large Pre-trained Neural Models

arXiv: https://arxiv.org/abs/2109.08914 · HTML: https://mlsanity.com/paper/2109.08914/

## Key facts

- Cited by 52 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 48 references that are in the map.
- First posted on arXiv 18 Sep 2021.

## Builds on (most cited first, 30 of 31)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models](https://mlsanity.com/paper/2009.11462/) (2020)
- [Neural Network Acceptability Judgments](https://mlsanity.com/paper/1805.12471/) (2018)
- [Plug and Play Language Models: A Simple Approach to Controlled Text Generation](https://mlsanity.com/paper/1912.02164/) (2019)
- [GeDi: Generative Discriminator Guided Sequence Generation](https://mlsanity.com/paper/2009.06367/) (2020)
- [Toward Controlled Generation of Text](https://mlsanity.com/paper/1703.00955/) (2017)
- [Style Transfer from Non-Parallel Text by Cross-Alignment](https://mlsanity.com/paper/1705.09655/) (2017)
- [Dear Sir or Madam, May I introduce the GYAFC Dataset: Corpus, Benchmarks and Metrics for Formality Style Transfer](https://mlsanity.com/paper/1803.06535/) (2018)
- [Delete, Retrieve, Generate: A Simple Approach to Sentiment and Style Transfer](https://mlsanity.com/paper/1804.06437/) (2018)
- [Reformulating Unsupervised Style Transfer as Paraphrase Generation](https://mlsanity.com/paper/2010.05700/) (2020)
- [SemEval-2020 Task 12: Multilingual Offensive Language Identification in Social Media (OffensEval 2020)](https://mlsanity.com/paper/2006.07235/) (2020)
- [Disentangled Representation Learning for Non-Parallel Text Style Transfer](https://mlsanity.com/paper/1808.04339/) (2018)
- [Style Transfer Through Back-Translation](https://mlsanity.com/paper/1804.09000/) (2018)
- [Conditional BERT Contextual Augmentation](https://mlsanity.com/paper/1812.06705/) (2018)
- [Beyond BLEU: Training Neural Machine Translation with Semantic Similarity](https://mlsanity.com/paper/1909.06694/) (2019)
- [Transforming Delete, Retrieve, Generate Approach for Controlled Text Style Transfer](https://mlsanity.com/paper/1908.09368/) (2019)
- [Politeness Transfer: A Tag and Generate Approach](https://mlsanity.com/paper/2004.14257/) (2020)
- [A Probabilistic Formulation of Unsupervised Text Style Transfer](https://mlsanity.com/paper/2002.03912/) (2020)
- [A Dual Reinforcement Learning Framework for Unsupervised Text Style Transfer](https://mlsanity.com/paper/1905.10060/) (2019)
- [Fighting Offensive Language on Social Media with Unsupervised Text Style Transfer](https://mlsanity.com/paper/1805.07685/) (2018)
- ["Mask and Infill" : Applying Masked Language Model to Sentiment Transfer](https://mlsanity.com/paper/1908.08039/) (2019)
- [ParaNMT-50M: Pushing the Limits of Paraphrastic Sentence Embeddings with Millions of Machine Translations](https://mlsanity.com/paper/1711.05732/) (2017)
- [Civil Rephrases Of Toxic Texts With Self-Supervised Transformers](https://mlsanity.com/paper/2102.05456/) (2021)
- [Style-transfer and Paraphrase: Looking for a Sensible Semantic Similarity Metric](https://mlsanity.com/paper/2004.05001/) (2020)
- [Stable Style Transformer: Delete and Generate Approach with Encoder-Decoder for Text Style Transfer](https://mlsanity.com/paper/2005.12086/) (2020)
- [Don't Patronize Me! An Annotated Dataset with Patronizing and Condescending Language towards Vulnerable Communities](https://mlsanity.com/paper/2011.08320/) (2020)
- [Towards A Friendly Online Community: An Unsupervised Style Transfer Framework for Profanity Redaction](https://mlsanity.com/paper/2011.00403/) (2020)
- [Towards non-toxic landscapes: Automatic toxic comment detection using DNN](https://mlsanity.com/paper/1911.08395/) (2019)

## Built on by (most cited first, 30 of 52)

- [A Survey on Large Language Model (LLM) Security and Privacy: The Good, the Bad, and the Ugly](https://mlsanity.com/paper/2312.02003/) (2023)
- [You Only Prompt Once: On the Capabilities of Prompt Learning on Large Language Models to Tackle Toxic Content](https://mlsanity.com/paper/2308.05596/) (2023)
- [Detoxifying Text with MaRCo: Controllable Revision with Experts and Anti-Experts](https://mlsanity.com/paper/2212.10543/) (2022)
- [Unified Detoxifying and Debiasing in Language Generation via Inference-time Adaptive Optimization](https://mlsanity.com/paper/2210.04492/) (2022)
- [ToViLaG: Your Visual-Language Generative Model is Also An Evildoer](https://mlsanity.com/paper/2312.11523/) (2023)
- [Goodtriever: Adaptive Toxicity Mitigation with Retrieval-augmented Models](https://mlsanity.com/paper/2310.07589/) (2023)
- [MERA: A Comprehensive LLM Evaluation in Russian](https://mlsanity.com/paper/2401.04531/) (2024)
- [Token-wise Influential Training Data Retrieval for Large Language Models](https://mlsanity.com/paper/2405.11724/) (2024)
- [ParaGuide: Guided Diffusion Paraphrasers for Plug-and-Play Textual Style Transfer](https://mlsanity.com/paper/2308.15459/) (2023)
- [From One to Many: Expanding the Scope of Toxicity Mitigation in Language Models](https://mlsanity.com/paper/2403.03893/) (2024)
- [TinyStyler: Efficient Few-Shot Text Style Transfer with Authorship Embeddings](https://mlsanity.com/paper/2406.15586/) (2024)
- [Walking in Others' Shoes: How Perspective-Taking Guides Large Language Models in Reducing Toxicity and Bias](https://mlsanity.com/paper/2407.15366/) (2024)
- [Exploring Methods for Cross-lingual Text Style Transfer: The Case of Text Detoxification](https://mlsanity.com/paper/2311.13937/) (2023)
- [Towards Safer Generative Language Models: A Survey on Safety Risks, Evaluations, and Improvements](https://mlsanity.com/paper/2302.09270/) (2023)
- [MultiParaDetox: Extending Text Detoxification with Parallel Data to New Languages](https://mlsanity.com/paper/2404.02037/) (2024)
- [Distilling Text Style Transfer With Self-Explanation From LLMs](https://mlsanity.com/paper/2403.01106/) (2024)
- [Multilingual and Explainable Text Detoxification with Parallel Corpora](https://mlsanity.com/paper/2412.11691/) (2024)
- [Language Detoxification with Attribute-Discriminative Latent Space](https://mlsanity.com/paper/2210.10329/) (2022)
- [Text Detoxification as Style Transfer in English and Hindi](https://mlsanity.com/paper/2402.07767/) (2024)
- [Target-Agnostic Gender-Aware Contrastive Learning for Mitigating Bias in Multilingual Machine Translation](https://mlsanity.com/paper/2305.14016/) (2023)
- [Learn What NOT to Learn: Towards Generative Safety in Chatbots](https://mlsanity.com/paper/2304.11220/) (2023)
- [IF-GUIDE: Influence Function-Guided Detoxification of LLMs](https://mlsanity.com/paper/2506.01790/) (2025)
- [Don't Take This Out of Context! On the Need for Contextual Models and Evaluations for Stylistic Rewriting](https://mlsanity.com/paper/2305.14755/) (2023)
- [DiffuDetox: A Mixed Diffusion Model for Text Detoxification](https://mlsanity.com/paper/2306.08505/) (2023)
- [CMD: a framework for Context-aware Model self-Detoxification](https://mlsanity.com/paper/2308.08295/) (2023)
- [MisgenderMender: A Community-Informed Approach to Interventions for Misgendering](https://mlsanity.com/paper/2404.14695/) (2024)
- [Generative Data Refinement: Just Ask for Better Data](https://mlsanity.com/paper/2509.08653/) (2025)
- [FrenchToxicityPrompts: a Large Benchmark for Evaluating and Mitigating Toxicity in French Texts](https://mlsanity.com/paper/2406.17566/) (2024)
- [The State of Profanity Obfuscation in Natural Language Processing](https://mlsanity.com/paper/2210.07595/) (2022)
- [Style Transfer with Multi-iteration Preference Optimization](https://mlsanity.com/paper/2406.11581/) (2024)

## Abstract (arXiv)

We present two novel unsupervised methods for eliminating toxicity in text. Our first method combines two recent ideas: (1) guidance of the generation process with small style-conditional language models and (2) use of paraphrasing models to perform style transfer. We use a well-performing paraphraser guided by style-trained language models to keep the text content and remove toxicity. Our second method uses BERT to replace toxic words with their non-offensive synonyms. We make the method more flexible by enabling BERT to replace mask tokens with a variable number of words. Finally, we present the first large-scale comparative study of style transfer models on the task of toxicity removal. We compare our models with a number of methods for style transfer. The models are evaluated in a reference-free way using a combination of unsupervised style transfer metrics. Both methods we suggest yield new SOTA results.