mlsanityData to 7 Oct 2026

ToxiGen: A Large-Scale Machine-Generated Dataset for Adversarial and Implicit Hate Speech Detection

arXiv 2203.09509 ↗First posted 17 Mar 2022cs.CLcited by 419

Not on the map. Why?

Cited by 55 in LLM safety and security, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Toxic language detection systems often falsely flag text that contains minority group mentions as toxic, as those groups are often the targets of online hate. Such over-reliance on spurious correlations also causes systems to struggle with detecting implicitly toxic language. To help mitigate these issues, we create ToxiGen, a new large-scale and machine-generated dataset of 274k toxic and benign statements about 13 minority groups. We develop a demonstration-based prompting framework and an adversarial classifier-in-the-loop decoding method to generate subtly toxic and benign text with a massive pretrained language model. Controlling machine generation in this way allows ToxiGen to cover implicitly toxic text at a larger scale, and about more demographic groups, than previous resources of human-written text. We conduct a human evaluation on a challenging subset of ToxiGen and find that annotators struggle to distinguish machine-generated text from human-written language. We also find that 94.5% of toxic examples are labeled as hate speech by human annotators. Using three publicly-available datasets, we show that finetuning a toxicity classifier on our data improves its performance on human-written data substantially. We also demonstrate that ToxiGen can be used to fight machine-generated toxicity as finetuning improves the classifier significantly on our evaluation subset. Our code and data can be found at https://github.com/microsoft/ToxiGen.

Key facts. Cited by 419 papers in the map (data through 7 Oct 2026). Builds on 33 of its 69 references that are in the map. First posted on arXiv 17 Mar 2022.

Builds on

33 of 69 references.

References · 33

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
  3. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,255
  4. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,235
  5. 2021What Makes Good In-Context Examples for GPT-3?cited by 1,184
  6. 2018Hierarchical Neural Story Generationcited by 1,182
  7. 2017Automated Hate Speech Detection and the Problem of Offensive Languagecited by 724
  8. 2019The Woman Worked as a Babysitter: On Biases in Language Generationcited by 540
  9. 2021DExperts: Decoding-Time Controlled Text Generation with Experts and Anti-Expertscited by 434
  10. 2019Social Bias Frames: Reasoning about Social and Power Implications of Languagecited by 377
  11. 2020GeDi: Generative Discriminator Guided Sequence Generationcited by 377
  12. 2021FUDGE: Controlled Text Generation With Future Discriminatorscited by 374
21 more
  1. 2021Dealing with Disagreements: Looking Beyond the Majority Vote in Subjective Annotationscited by 306
  2. 2021All That's 'Human' Is Not Gold: Evaluating Human Evaluation of Generated Textcited by 287
  3. 2019Predicting the Type and Target of Offensive Posts in Social Mediacited by 268
  4. 2018Large Scale Crowdsourcing and Characterization of Twitter Abusive Behaviorcited by 265
  5. 2021Annotators with Attitudes: How Annotator Beliefs And Identities Bias Toxic Language Detectioncited by 258
  6. 2017Lexically Constrained Decoding for Sequence Generation Using Grid Beam Searchcited by 234
  7. 2020Learning from the Worst: Dynamically Generated Datasets to Improve Online Hate Detectioncited by 217
  8. 2021Latent Hatred: A Benchmark for Understanding Implicit Hate Speechcited by 216
  9. 2020HateCheck: Functional Tests for Hate Speech Detection Modelscited by 215
  10. 2020HateBERT: Retraining BERT for Abusive Language Detection in Englishcited by 206
  11. 2018Hate Speech Dataset from a White Supremacy Forumcited by 196
  12. 2019Build it Break it Fix it for Dialogue Safety: Robustness from Adversarial Human Attackcited by 176
  13. 2020NeuroLogic Decoding: (Un)supervised Neural Text Generation with Predicate Logic Constraintscited by 146
  14. 2018Learning to Write with Cooperative Discriminatorscited by 119
  15. 2021Detoxifying Language Models Risks Marginalizing Minority Voicescited by 106
  16. 2016Guided Open Vocabulary Image Captioning with Constrained Beam Searchcited by 105
  17. 2017Measuring the Reliability of Hate Speech Annotations: The Case of the European Refugee Crisiscited by 95
  18. 2021Challenges in Automated Debiasing for Toxic Language Detectioncited by 95
  19. 2021Agreeing to Disagree: Annotating Offensive Language Datasets with Annotators' Disagreementcited by 71
  20. 2018The Effect of Extremist Violence on Hateful Speech Onlinecited by 17
  21. 2021TweetBLM: A Hate Speech Dataset and Analysis of Black Lives Matter-related Microblogs on Twittercited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 18; 2023: 83; 2024: 124; 2025: 113; 2026: 81.

419 papers.

Sort:

By area:

  1. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  2. 2023Code Llama: Open Foundation Models for Codecited by 2,105
  3. 2022Discovering Language Model Behaviors with Model-Written Evaluationscited by 805
  4. 2023AGIEval: A Human-Centric Benchmark for Evaluating Foundation Modelscited by 701
  5. 2023Baichuan 2: Open Large-scale Language Modelscited by 664
  6. 2023Textbooks Are All You Need II: phi-1.5 technical reportcited by 559
  7. 2023XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Modelscited by 477
  8. 2023SmoothLLM: Defending Large Language Models Against Jailbreaking Attackscited by 434
  9. 2023Towards Measuring the Representation of Subjective Global Opinions in Language Modelscited by 424
  10. 2023Orca: Progressive Learning from Complex Explanation Traces of GPT-4cited by 369
  11. 2022A Holistic Approach to Undesired Content Detection in the Real Worldcited by 337
  12. 2023Visual Adversarial Examples Jailbreak Aligned Large Language Modelscited by 307
38 more
  1. 2023Effective Long-Context Scaling of Foundation Modelscited by 301
  2. 2024Aya Model: An Instruction Finetuned Open-Access Multilingual Language Modelcited by 292
  3. 2023Shadow Alignment: The Ease of Subverting Safely-Aligned Language Modelscited by 284
  4. 2022Aging with GRACE: Lifelong Model Editing with Discrete Key-Value Adaptorscited by 267
  5. 2023MM-SafetyBench: A Benchmark for Safety Evaluation of Multimodal Large Language Modelscited by 255
  6. 2023Large Language Model Alignment: A Surveycited by 241
  7. 2024On LLMs-Driven Synthetic Data Generation, Curation, and Evaluation: A Surveycited by 240
  8. 2024SALAD-Bench: A Hierarchical and Comprehensive Safety Benchmark for Large Language Modelscited by 225
  9. 2023Multilingual Jailbreak Challenges in Large Language Modelscited by 222
  10. 2024OR-Bench: An Over-Refusal Benchmark for Large Language Modelscited by 222
  11. 2023A Pretrainer's Guide to Training Data: Measuring the Effects of Data Age, Domain Coverage, Quality, & Toxicitycited by 180
  12. 2023Do-Not-Answer: A Dataset for Evaluating Safeguards in LLMscited by 176
  13. 2024RM-Bench: Benchmarking Reward Models of Language Models with Subtlety and Stylecited by 168
  14. 2023Understanding Social Reasoning in Language Models with Language Modelscited by 166
  15. 2024PrivacyLens: Evaluating Privacy Norm Awareness of Language Models in Actioncited by 161
  16. 2025Phi-4-reasoning Technical Reportcited by 146
  17. 2023Almanac: Retrieval-Augmented Language Models for Clinical Medicinecited by 122
  18. 2023Increasing Diversity While Maintaining Accuracy: Text Data Generation with Large Language Models and Human Interventionscited by 121
  19. 2024Know Your Limits: A Survey of Abstention in Large Language Modelscited by 116
  20. 2022Pile of Law: Learning Responsible Data Filtering from the Law and a 256GB Open-Source Legal Datasetcited by 113
  21. 2025A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deploymentcited by 110
  22. 2025Large Language Models Often Know When They Are Being Evaluatedcited by 105
  23. 2024Large Language Models Meet NLP: A Surveycited by 102
  24. 2023EasyEdit: An Easy-to-use Knowledge Editing Framework for Large Language Modelscited by 97
  25. 2023Revisiting Out-of-distribution Robustness in NLP: Benchmark, Analysis, and LLMs Evaluationscited by 96
  26. 2023Synthetic Data Generation with Large Language Models for Text Classification: Potential and Limitationscited by 96
  27. 2023Evaluating the Social Impact of Generative AI Systems in Systems and Societycited by 91
  28. 2024RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMscited by 89
  29. 2024Med42-v2: A Suite of Clinical LLMscited by 89
  30. 2024Steering Language Model Refusal with Sparse Autoencoderscited by 89
  31. 2024Tuning Language Models by Proxycited by 88
  32. 2024Curiosity-driven Red-teaming for Large Language Modelscited by 83
  33. 2023FLIRT: Feedback Loop In-context Red Teamingcited by 81
  34. 2023A Survey of GPT-3 Family Large Language Models Including ChatGPT and GPT-4cited by 81
  35. 2023Use of LLMs for Illicit Purposes: Threats, Prevention Measures, and Vulnerabilitiescited by 76
  36. 2024SafetyPrompts: a Systematic Review of Open Datasets for Evaluating and Improving Large Language Model Safetycited by 75
  37. 2023SimpleSafetyTests: a Test Suite for Identifying Critical Safety Risks in Large Language Modelscited by 72
  38. 2023Grounded Decoding: Guiding Text Generation with Grounded Models for Embodied Agentscited by 66

50 most cited of 419 All 419 (JSON)