mlsanityData to 7 Oct 2026

Adding Instructions during Pretraining: Effective Way of Controlling Toxicity in Language Models

arXiv 2302.07388 ↗First posted 14 Feb 2023cs.CL cs.AIcited by 16

Not on the map. Why?

Abstract (arXiv)

Pretrained large language models have become indispensable for solving various natural language processing (NLP) tasks. However, safely deploying them in real world applications is challenging because they generate toxic content. To address this challenge, we propose two novel pretraining data augmentation strategies that significantly reduce model toxicity without compromising its utility. Our two strategies are: (1) MEDA: adds raw toxicity score as meta-data to the pretraining samples, and (2) INST: adds instructions to those samples indicating their toxicity. Our results indicate that our best performing strategy (INST) substantially reduces the toxicity probability up to 61% while preserving the accuracy on five benchmark NLP tasks as well as improving AUC scores on four bias detection tasks by 1.3%. We also demonstrate the generalizability of our techniques by scaling the number of training samples and the number of model parameters.

Key facts. Cited by 16 papers in the map (data through 7 Oct 2026). Builds on 33 of its 41 references that are in the map. First posted on arXiv 14 Feb 2023.

Builds on

33 of 41 references.

References · 33

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  5. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  6. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  7. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  8. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  9. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  10. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,548
  11. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,235
  12. 2022Red Teaming Language Models with Language Modelscited by 983
21 more
  1. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 960
  2. 2019Adversarial NLI: A New Benchmark for Natural Language Understandingcited by 798
  3. 2019Plug and Play Language Models: A Simple Approach to Controlled Text Generationcited by 793
  4. 2019CTRL: A Conditional Transformer Language Model for Controllable Generationcited by 754
  5. 2022Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Modelcited by 557
  6. 2019Defending Against Neural Fake Newscited by 553
  7. 2021DExperts: Decoding-Time Controlled Text Generation with Experts and Anti-Expertscited by 434
  8. 2019Social Bias Frames: Reasoning about Social and Power Implications of Languagecited by 377
  9. 2020GeDi: Generative Discriminator Guided Sequence Generationcited by 377
  10. 2021Self-Diagnosis and Self-Debiasing: A Proposal for Reducing Corpus-Based Bias in NLPcited by 336
  11. 2017The E2E Dataset: New Challenges For End-to-End Generationcited by 252
  12. 2018A Simple Method for Commonsense Reasoningcited by 223
  13. 2019WinoGrande: An Adversarial Winograd Schema Challenge at Scalecited by 194
  14. 2021Process for Adapting Language Models to Society (PALMS) with Values-Targeted Datasetscited by 179
  15. 2021Challenges in Detoxifying Language Modelscited by 169
  16. 2021Detoxifying Language Models Risks Marginalizing Minority Voicescited by 106
  17. 2020The Radicalization Risks of GPT-3 and Advanced Neural Language Modelscited by 76
  18. 2022Exploring the Limits of Domain-Adaptive Training for Detoxifying Large-Scale Language Modelscited by 67
  19. 2021Mitigating harm in language models with conditional-likelihood filtrationcited by 38
  20. 2021Few-shot Instruction Prompts for Pretrained Language Models to Detect Social Biasescited by 21
  21. 2020Case Study: Deontological Ethics in NLPcited by 12

Built on by

Cited by year

Citing papers per year: 2023: 5; 2024: 6; 2025: 5.

16 papers.

Sort:

By area:

  1. 2023Scaling Data-Constrained Language Modelscited by 391
  2. 2024Detoxifying Large Language Models via Knowledge Editingcited by 86
  3. 2023Self-Detoxifying Language Models via Toxification Reversalcited by 45
  4. 2024Securing Large Language Models: Threats, Vulnerabilities and Responsible Practicescited by 36
  5. 2025A Comprehensive Analysis of Large Language Model Outputs: Similarity, Diversity, and Biascited by 10
  6. 2023Transformation vs Tradition: Artificial General Intelligence (AGI) for Arts and Humanitiescited by 9
  7. 2025IF-GUIDE: Influence Function-Guided Detoxification of LLMscited by 7
  8. 2023CMD: a framework for Context-aware Model self-Detoxificationcited by 6
  9. 2024Adversarial DPO: Harnessing Harmful Data for Reducing Toxicity with Minimal Impact on Coherence and Evasiveness in Dialogue Agentscited by 6
  10. 2024Can LLMs be Fooled? Investigating Vulnerabilities in LLMscited by 5
  11. 2024Multi-Objective Linguistic Control of Large Language Modelscited by 4
  12. 2025Multi-level Value Alignment in Agentic AI Systems: Survey and Perspectivescited by 4
4 more
  1. 2025TTS-1 Technical Reportcited by 3
  2. 2023Automatic Construction of a Korean Toxic Instruction Dataset for Ethical Tuning of Large Language Modelscited by 1
  3. 2025Mimicking How Humans Interpret Out-of-Context Sentences Through Controlled Toxicity Decodingcited by 1
  4. 2024Mitigating Biases to Embrace Diversity: A Comprehensive Annotation Benchmark for Toxic Languagecited by 0