mlsanityData to 7 Oct 2026

Red Teaming Language Models with Language Models

arXiv 2202.03286 ↗First posted 7 Feb 2022cs.CL cs.AI cs.CRcited by 983

On the map: LLM safety and security → Prompt injection and agent security

Abstract (arXiv)

Language Models (LMs) often cannot be deployed because of their potential to harm users in hard-to-predict ways. Prior work identifies harmful behaviors before deployment by using human annotators to hand-write test cases. However, human annotation is expensive, limiting the number and diversity of test cases. In this work, we automatically find cases where a target LM behaves in a harmful way, by generating test cases ("red teaming") using another LM. We evaluate the target LM's replies to generated test questions using a classifier trained to detect offensive content, uncovering tens of thousands of offensive replies in a 280B parameter LM chatbot. We explore several methods, from zero-shot generation to reinforcement learning, for generating test cases with varying levels of diversity and difficulty. Furthermore, we use prompt engineering to control LM-generated test cases to uncover a variety of other harms, automatically finding groups of people that the chatbot discusses in offensive ways, personal and hospital phone numbers generated as the chatbot's own contact info, leakage of private training data in generated text, and harms that occur over the course of a conversation. Overall, LM-based red teaming is one promising tool (among many needed) for finding and fixing diverse, undesirable LM behaviors before impacting users.

Key facts. Cited by 983 papers in the map (data through 7 Oct 2026). Placed in Prompt injection and agent security, one of 12 directions in LLM safety and security. Builds on 77 of its 109 references that are in the map. First posted on arXiv 7 Feb 2022.

Builds on

77 of 109 references.

References · 77

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  4. 2014Explaining and Harnessing Adversarial Examplescited by 7,287
  5. 2013Intriguing properties of neural networkscited by 5,114
  6. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  7. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  8. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  9. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  10. 2016Deep Learning with Differential Privacycited by 2,682
  11. 2020Learning to summarize from human feedbackcited by 2,562
  12. 2016Membership Inference Attacks against Machine Learning Modelscited by 2,221
38 more
  1. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,218
  2. 2020Extracting Training Data from Large Language Modelscited by 2,144
  3. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  4. 2016Concrete Problems in AI Safetycited by 1,582
  5. 2016Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddingscited by 1,304
  6. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,235
  7. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
  8. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,023
  9. 2020Beyond Accuracy: Behavioral Testing of NLP models with CheckListcited by 979
  10. 2021Ethical and social risks of harm from Language Modelscited by 957
  11. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 896
  12. 2016Bag of Tricks for Efficient Text Classificationcited by 869
  13. 2018The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networkscited by 847
  14. 2019Adversarial NLI: A New Benchmark for Natural Language Understandingcited by 798
  15. 2019Universal Adversarial Triggers for Attacking and Analyzing NLPcited by 709
  16. 2017Adversarial Examples for Evaluating Reading Comprehension Systemscited by 568
  17. 2019The Woman Worked as a Babysitter: On Biases in Language Generationcited by 540
  18. 2016Delving into Transferable Adversarial Examples and Black-box Attackscited by 530
  19. 2019Neural Text Generation with Unlikelihood Trainingcited by 489
  20. 2018Texygen: A Benchmarking Platform for Text Generation Modelscited by 484
  21. 2016Transferability in Machine Learning: from Phenomena to Black-Box Attacks using Adversarial Samplescited by 439
  22. 2021BOLD: Dataset and Metrics for Measuring Biases in Open-Ended Language Generationcited by 410
  23. 2015Deep Learning with Limited Numerical Precisioncited by 368
  24. 2016How NOT To Evaluate Your Dialogue System: An Empirical Study of Unsupervised Evaluation Metrics for Dialogue Response Generationcited by 356
  25. 2021True Few-Shot Learning with Language Modelscited by 343
  26. 2021Self-Diagnosis and Self-Debiasing: A Proposal for Reducing Corpus-Based Bias in NLPcited by 336
  27. 2021Unsolved Problems in ML Safetycited by 291
  28. 2021Recursively Summarizing Books with Human Feedbackcited by 287
  29. 2019Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialogcited by 283
  30. 2019Adversarial Attacks and Defenses in Images, Graphs and Text: A Reviewcited by 234
  31. 2021Polyjuice: Generating Counterfactuals for Explaining, Evaluating, and Improving Modelscited by 219
  32. 2020HateCheck: Functional Tests for Hate Speech Detection Modelscited by 215
  33. 2017Generating Natural Adversarial Examplescited by 182
  34. 2021Process for Adapting Language Models to Society (PALMS) with Values-Targeted Datasetscited by 179
  35. 2019Build it Break it Fix it for Dialogue Safety: Robustness from Adversarial Human Attackcited by 176
  36. 2021Challenges in Detoxifying Language Modelscited by 169
  37. 2018Counterfactual Fairness in Text Classification through Robustnesscited by 150
  38. 2019Reducing Sentiment Bias in Language Models via Counterfactual Evaluationcited by 148

50 most cited of 77 All 77 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 46; 2023: 159; 2024: 234; 2025: 225; 2026: 319.

983 papers.

Sort:

By area:

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  3. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  4. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  5. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,748
  6. 2023A Survey of Large Language Modelscited by 2,537
  7. 2022Emergent Abilities of Large Language Modelscited by 2,483
  8. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,565
  9. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,330
  10. 2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,326
  11. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!cited by 1,106
  12. 2024Refusal in Language Models Is Mediated by a Single Directioncited by 960
38 more
  1. 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Datasetcited by 828
  2. 2022Discovering Language Model Behaviors with Model-Written Evaluationscited by 805
  3. 2023StarCoder: may the source be with you!cited by 790
  4. 2023Whose Opinions Do Language Models Reflect?cited by 759
  5. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 743
  6. 2023AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Modelscited by 738
  7. 2022Discovering Latent Knowledge in Language Models Without Supervisioncited by 733
  8. 2023Can Large Language Models Be an Alternative to Human Evaluations?cited by 729
  9. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learnedcited by 702
  10. 2023A Comprehensive Overview of Large Language Modelscited by 639
  11. 2022Improving alignment of dialogue agents via targeted human judgementscited by 527
  12. 2023Identifying the Risks of LM Agents with an LM-Emulated Sandboxcited by 514
  13. 2024JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Modelscited by 494
  14. 2023Reinforced Self-Training (ReST) for Language Modelingcited by 486
  15. 2024Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Trainingcited by 478
  16. 2023XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Modelscited by 477
  17. 2023Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervisioncited by 445
  18. 2024The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructionscited by 441
  19. 2023Catastrophic Jailbreak of Open-source LLMs via Exploiting Generationcited by 439
  20. 2023Towards Measuring the Representation of Subjective Global Opinions in Language Modelscited by 424
  21. 2023GPT-4 Is Too Smart To Be Safe: Stealthy Chat with LLMs via Ciphercited by 401
  22. 2023Jailbreak and Guard Aligned Language Models with Only Few In-Context Demonstrationscited by 401
  23. 2023UltraFeedback: Boosting Language Models with Scaled AI Feedbackcited by 394
  24. 2023Understanding the Effects of RLHF on LLM Generalisation and Diversitycited by 386
  25. 2023Towards Expert-Level Medical Question Answering with Large Language Modelscited by 382
  26. 2022Self-critiquing models for assisting human evaluatorscited by 372
  27. 2023FigStep: Jailbreaking Large Vision-Language Models via Typographic Visual Promptscited by 368
  28. 2022RLPrompt: Optimizing Discrete Text Prompts with Reinforcement Learningcited by 361
  29. 2024A StrongREJECT for Empty Jailbreakscited by 352
  30. 2023NeMo Guardrails: A Toolkit for Controllable and Safe LLM Applications with Programmable Railscited by 347
  31. 2022A Holistic Approach to Undesired Content Detection in the Real Worldcited by 337
  32. 2023Principle-Driven Self-Alignment of Language Models from Scratch with Minimal Human Supervisioncited by 318
  33. 2023Visual Adversarial Examples Jailbreak Aligned Large Language Modelscited by 307
  34. 2024Improving Alignment and Robustness with Circuit Breakerscited by 302
  35. 2022Teaching language models to support answers with verified quotescited by 290
  36. 2023Low-Resource Languages Jailbreak GPT-4cited by 290
  37. 2023On Evaluating Adversarial Robustness of Large Vision-Language Modelscited by 281
  38. 2022Prompting GPT-3 To Be Reliablecited by 258

50 most cited of 983 All 983 (JSON)