mlsanityData to 7 Oct 2026

RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models

arXiv 2009.11462 ↗First posted 24 Sep 2020cs.CLcited by 1,235

Not on the map. Why?

Cited by 215 in LLM safety and security, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Pretrained neural language models (LMs) are prone to generating racist, sexist, or otherwise toxic language which hinders their safe deployment. We investigate the extent to which pretrained LMs can be prompted to generate toxic language, and the effectiveness of controllable text generation algorithms at preventing such toxic degeneration. We create and release RealToxicityPrompts, a dataset of 100K naturally occurring, sentence-level prompts derived from a large corpus of English web text, paired with toxicity scores from a widely-used toxicity classifier. Using RealToxicityPrompts, we find that pretrained LMs can degenerate into toxic text even from seemingly innocuous prompts. We empirically assess several controllable generation methods, and find that while data- or compute-intensive methods (e.g., adaptive pretraining on non-toxic data) are more effective at steering away from toxicity than simpler solutions (e.g., banning "bad" words), no current method is failsafe against neural toxic degeneration. To pinpoint the potential cause of such persistent toxic degeneration, we analyze two web text corpora used to pretrain several LMs (including GPT-2; Radford et. al, 2019), and find a significant amount of offensive, factually unreliable, and otherwise toxic content. Our work provides a test bed for evaluating toxic generations by LMs and stresses the need for better data selection processes for pretraining.

Key facts. Cited by 1,235 papers in the map (data through 7 Oct 2026). First posted on arXiv 24 Sep 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 54; 2022: 126; 2023: 255; 2024: 324; 2025: 256; 2026: 220.

1,235 papers.

Sort:

By area:

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  5. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  6. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  7. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  8. 2023A Survey of Large Language Modelscited by 2,537
  9. 2022Emergent Abilities of Large Language Modelscited by 2,483
  10. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
  11. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,565
  12. 2023Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injectioncited by 1,339
38 more
  1. 2021Improving language models by retrieving from trillions of tokenscited by 1,278
  2. 2023A Survey on Evaluation of Large Language Modelscited by 1,227
  3. 2022LaMDA: Language Models for Dialog Applicationscited by 1,218
  4. 2023PaLM 2 Technical Reportcited by 1,154
  5. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!cited by 1,106
  6. 2022Editing Models with Task Arithmeticcited by 1,076
  7. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,023
  8. 2022Red Teaming Language Models with Language Modelscited by 983
  9. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 890
  10. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 837
  11. 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Datasetcited by 828
  12. 2023Steering Language Models With Activation Engineeringcited by 824
  13. 2022Discovering Language Model Behaviors with Model-Written Evaluationscited by 805
  14. 2023StarCoder: may the source be with you!cited by 790
  15. 2022Ignore Previous Prompt: Attack Techniques For Language Modelscited by 760
  16. 2023CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquingcited by 751
  17. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 743
  18. 2022Galactica: A Large Language Model for Sciencecited by 736
  19. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learnedcited by 702
  20. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignmentcited by 672
  21. 2023A Comprehensive Overview of Large Language Modelscited by 639
  22. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedbackcited by 630
  23. 2023Bias and Fairness in Large Language Models: A Surveycited by 626
  24. 2023Instruction Tuning for Large Language Models: A Surveycited by 617
  25. 2023Eliciting Latent Predictions from Transformers with the Tuned Lenscited by 579
  26. 2023Prompt Injection attack against LLM-integrated Applicationscited by 568
  27. 2022Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Modelcited by 557
  28. 2024ORPO: Monolithic Preference Optimization without Reference Modelcited by 556
  29. 2024How Johnny Can Persuade LLMs to Jailbreak Them: Rethinking Persuasion to Challenge AI Safety by Humanizing LLMscited by 533
  30. 2022Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Spacecited by 522
  31. 2023GPTFUZZER: Red Teaming Large Language Models with Auto-Generated Jailbreak Promptscited by 520
  32. 2022Safe Latent Diffusion: Mitigating Inappropriate Degeneration in Diffusion Modelscited by 501
  33. 2024The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearningcited by 488
  34. 2021Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpuscited by 481
  35. 2023XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Modelscited by 477
  36. 2022MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledgecited by 469
  37. 2024WildGuard: Open One-Stop Moderation Tools for Safety Risks, Jailbreaks, and Refusals of LLMscited by 460
  38. 2021Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Surveycited by 457

50 most cited of 1,235 All 1,235 (JSON)