# Exploring the Limits of Domain-Adaptive Training for Detoxifying Large-Scale Language Models

arXiv: https://arxiv.org/abs/2202.04173 · HTML: https://mlsanity.com/paper/2202.04173/

## Key facts

- Cited by 68 papers in the map (data through 14 Oct 2026).
- Builds on 45 of its 65 references that are in the map.
- First posted on arXiv 8 Feb 2022.

## Builds on (most cited first, 30 of 45)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [Parameter-Efficient Transfer Learning for NLP](https://mlsanity.com/paper/1902.00751/) (2019)
- [HellaSwag: Can a Machine Really Finish Your Sentence?](https://mlsanity.com/paper/1905.07830/) (2019)
- [Prefix-Tuning: Optimizing Continuous Prompts for Generation](https://mlsanity.com/paper/2101.00190/) (2021)
- [The Power of Scale for Parameter-Efficient Prompt Tuning](https://mlsanity.com/paper/2104.08691/) (2021)
- [Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity](https://mlsanity.com/paper/2101.03961/) (2021)
- [The Curious Case of Neural Text Degeneration](https://mlsanity.com/paper/1904.09751/) (2019)
- [PIQA: Reasoning about Physical Commonsense in Natural Language](https://mlsanity.com/paper/1911.11641/) (2019)
- [Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing](https://mlsanity.com/paper/2107.13586/) (2021)
- [Extracting Training Data from Large Language Models](https://mlsanity.com/paper/2012.07805/) (2020)
- [BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions](https://mlsanity.com/paper/1905.10044/) (2019)
- [Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism](https://mlsanity.com/paper/1909.08053/) (2019)
- [Don't Stop Pretraining: Adapt Language Models to Domains and Tasks](https://mlsanity.com/paper/2004.10964/) (2020)
- [Calibrate Before Use: Improving Few-Shot Performance of Language Models](https://mlsanity.com/paper/2102.09690/) (2021)
- [RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models](https://mlsanity.com/paper/2009.11462/) (2020)
- [A Diversity-Promoting Objective Function for Neural Conversation Models](https://mlsanity.com/paper/1510.03055/) (2015)
- [Red Teaming Language Models with Language Models](https://mlsanity.com/paper/2202.03286/) (2022)
- [The LAMBADA dataset: Word prediction requiring a broad discourse context](https://mlsanity.com/paper/1606.06031/) (2016)
- [Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inference](https://mlsanity.com/paper/2001.07676/) (2020)
- [RACE: Large-scale ReAding Comprehension Dataset From Examinations](https://mlsanity.com/paper/1704.04683/) (2017)
- [Adversarial NLI: A New Benchmark for Natural Language Understanding](https://mlsanity.com/paper/1910.14599/) (2019)
- [Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks](https://mlsanity.com/paper/1506.03099/) (2015)
- [Sequence-Level Knowledge Distillation](https://mlsanity.com/paper/1606.07947/) (2016)
- [Plug and Play Language Models: A Simple Approach to Controlled Text Generation](https://mlsanity.com/paper/1912.02164/) (2019)
- [Universal Adversarial Triggers for Attacking and Analyzing NLP](https://mlsanity.com/paper/1908.07125/) (2019)
- [It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learners](https://mlsanity.com/paper/2009.07118/) (2020)
- [Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model](https://mlsanity.com/paper/2201.11990/) (2022)

## Built on by (most cited first, 30 of 68)

- [Jailbroken: How Does LLM Safety Training Fail?](https://mlsanity.com/paper/2307.02483/) (2023)
- [GPT-4 Is Too Smart To Be Safe: Stealthy Chat with LLMs via Cipher](https://mlsanity.com/paper/2308.06463/) (2023)
- [A Wolf in Sheep's Clothing: Generalized Nested Jailbreak Prompts can Fool Large Language Models Easily](https://mlsanity.com/paper/2311.08268/) (2023)
- [Pretraining Language Models with Human Preferences](https://mlsanity.com/paper/2302.08582/) (2023)
- [In-context Vectors: Making In Context Learning More Effective and Controllable Through Latent Space Steering](https://mlsanity.com/paper/2311.06668/) (2023)
- [Factuality Enhanced Language Models for Open-Ended Text Generation](https://mlsanity.com/paper/2206.04624/) (2022)
- [GuardReasoner: Towards Reasoning-based LLM Safeguards](https://mlsanity.com/paper/2501.18492/) (2025)
- [Language Generation Models Can Cause Harm: So What Can We Do About It? An Actionable Survey](https://mlsanity.com/paper/2210.07700/) (2022)
- [BadChain: Backdoor Chain-of-Thought Prompting for Large Language Models](https://mlsanity.com/paper/2401.12242/) (2024)
- [FlipAttack: Jailbreak LLMs via Flipping](https://mlsanity.com/paper/2410.02832/) (2024)
- [Risk Taxonomy, Mitigation, and Assessment Benchmarks of Large Language Model Systems](https://mlsanity.com/paper/2401.05778/) (2024)
- [Safety in Large Reasoning Models: A Survey](https://mlsanity.com/paper/2504.17704/) (2025)
- [Fairness in Large Language Models: A Taxonomic Survey](https://mlsanity.com/paper/2404.01349/) (2024)
- [All Languages Matter: On the Multilingual Safety of Large Language Models](https://mlsanity.com/paper/2310.00905/) (2023)
- [Self-Detoxifying Language Models via Toxification Reversal](https://mlsanity.com/paper/2310.09573/) (2023)
- [Stress-Testing Capability Elicitation With Password-Locked Models](https://mlsanity.com/paper/2405.19550/) (2024)
- [On the Challenges of Using Black-Box APIs for Toxicity Evaluation in Research](https://mlsanity.com/paper/2304.12397/) (2023)
- [Click: Controllable Text Generation with Sequence Likelihood Contrastive Learning](https://mlsanity.com/paper/2306.03350/) (2023)
- [Towards Safe Self-Distillation of Internet-Scale Text-to-Image Diffusion Models](https://mlsanity.com/paper/2307.05977/) (2023)
- [Model Editing as a Robust and Denoised variant of DPO: A Case Study on Toxicity](https://mlsanity.com/paper/2405.13967/) (2024)
- [Unified Detoxifying and Debiasing in Language Generation via Inference-time Adaptive Optimization](https://mlsanity.com/paper/2210.04492/) (2022)
- [Get more for less: Principled Data Selection for Warming Up Fine-Tuning in LLMs](https://mlsanity.com/paper/2405.02774/) (2024)
- [ToViLaG: Your Visual-Language Generative Model is Also An Evildoer](https://mlsanity.com/paper/2312.11523/) (2023)
- [Goodtriever: Adaptive Toxicity Mitigation with Retrieval-augmented Models](https://mlsanity.com/paper/2310.07589/) (2023)
- [A Pseudo-Semantic Loss for Autoregressive Models with Logical Constraints](https://mlsanity.com/paper/2312.03905/) (2023)
- [Parameter-Efficient Detoxification with Contrastive Decoding](https://mlsanity.com/paper/2401.06947/) (2024)
- [From One to Many: Expanding the Scope of Toxicity Mitigation in Language Models](https://mlsanity.com/paper/2403.03893/) (2024)
- [Efficient LLM Jailbreak via Adaptive Dense-to-sparse Constrained Optimization](https://mlsanity.com/paper/2405.09113/) (2024)
- [Walking in Others' Shoes: How Perspective-Taking Guides Large Language Models in Reducing Toxicity and Bias](https://mlsanity.com/paper/2407.15366/) (2024)
- [Adding Instructions during Pretraining: Effective Way of Controlling Toxicity in Language Models](https://mlsanity.com/paper/2302.07388/) (2023)

## Abstract (arXiv)

Pre-trained language models (LMs) are shown to easily generate toxic language. In this work, we systematically explore domain-adaptive training to reduce the toxicity of language models. We conduct this study on three dimensions: training corpus, model size, and parameter efficiency. For the training corpus, we propose to leverage the generative power of LMs and generate nontoxic datasets for domain-adaptive training, which mitigates the exposure bias and is shown to be more data-efficient than using a curated pre-training corpus. We demonstrate that the self-generation method consistently outperforms the existing baselines across various model sizes on both automatic and human evaluations, even when it uses a 1/3 smaller training corpus. We then comprehensively study detoxifying LMs with parameter sizes ranging from 126M up to 530B (3x larger than GPT-3), a scale that has never been studied before. We find that i) large LMs have similar toxicity levels as smaller ones given the same pre-training corpus, and ii) large LMs require more endeavor to detoxify. We also explore parameter-efficient training methods for detoxification. We demonstrate that adding and training adapter-only layers in LMs not only saves a lot of parameters but also achieves a better trade-off between toxicity and perplexity than whole model adaptation for the large-scale models.