mlsanityData to 7 Oct 2026

Explore, Establish, Exploit: Red Teaming Language Models from Scratch

arXiv 2306.09442 ↗First posted 15 Jun 2023cs.CL cs.AI cs.LGcited by 111

On the map: LLM safety and security → Prompt injection and agent security

Abstract (arXiv)

Deploying large language models (LMs) can pose hazards from harmful outputs such as toxic or false text. Prior work has introduced automated tools that elicit harmful outputs to identify these risks. While this is a valuable step toward securing models, these approaches rely on a pre-existing way to efficiently classify undesirable outputs. Using a pre-existing classifier does not allow for red-teaming to be tailored to the target model. Furthermore, when failures can be easily classified in advance, red-teaming has limited marginal value because problems can be avoided by simply filtering training data and/or model outputs. Here, we consider red-teaming "from scratch," in which the adversary does not begin with a way to classify failures. Our framework consists of three steps: 1) Exploring the model's range of behaviors in the desired context; 2) Establishing a definition and measurement for undesired behavior (e.g., a classifier trained to reflect human evaluations); and 3) Exploiting the model's flaws using this measure to develop diverse adversarial prompts. We use this approach to red-team GPT-3 to discover classes of inputs that elicit false statements. In doing so, we construct the CommonClaim dataset of 20,000 statements labeled by humans as common-knowledge-true, common knowledge-false, or neither. We are making code and data available.

Key facts. Cited by 111 papers in the map (data through 7 Oct 2026). Placed in Prompt injection and agent security, one of 12 directions in LLM safety and security. Builds on 51 of its 72 references that are in the map. First posted on arXiv 15 Jun 2023.

Builds on

51 of 72 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023Multi-step Jailbreaking Privacy Attacks on ChatGPT11 Apr
  2. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned23 Aug
  3. 2022Red Teaming Language Models with Language Models7 Feb

Other references · 48

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  3. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  4. 2017Deep reinforcement learning from human preferencescited by 3,846
  5. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  6. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  7. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  8. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  9. 2022Survey of Hallucination in Natural Language Generationcited by 2,056
  10. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,722
  11. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  12. 2018FEVER: a large-scale dataset for Fact Extraction and VERificationcited by 1,362
36 more
  1. 2020On Faithfulness and Factuality in Abstractive Summarizationcited by 895
  2. 2023ChatGPT Outperforms Crowd-Workers for Text-Annotation Taskscited by 849
  3. 2022Discovering Language Model Behaviors with Model-Written Evaluationscited by 805
  4. 2023Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generationcited by 795
  5. 2023Whose Opinions Do Language Models Reflect?cited by 759
  6. 2022Discovering Latent Knowledge in Language Models Without Supervisioncited by 733
  7. 2022Teaching Models to Express Their Uncertainty in Wordscited by 731
  8. 2019Universal Adversarial Triggers for Attacking and Analyzing NLPcited by 709
  9. 2021Retrieval Augmentation Reduces Hallucination in Conversationcited by 691
  10. 2020KILT: a Benchmark for Knowledge Intensive Language Taskscited by 571
  11. 2023Jailbreaking ChatGPT via Prompt Engineering: An Empirical Studycited by 478
  12. 2018TextBugger: Generating Adversarial Text Against Real-world Applicationscited by 376
  13. 2022RLPrompt: Optimizing Discrete Text Prompts with Reinforcement Learningcited by 361
  14. 2023Hard Prompts Made Easy: Gradient-Based Discrete Optimization for Prompt Tuning and Discoverycited by 329
  15. 2022Teaching language models to support answers with verified quotescited by 290
  16. 2023Pretraining Language Models with Human Preferencescited by 245
  17. 2021Gradient-based Adversarial Attacks against Text Transformerscited by 233
  18. 2021Muppet: Massive Multi-task Representations with Pre-Finetuningcited by 192
  19. 2023Automatically Auditing Large Language Models via Discrete Optimizationcited by 187
  20. 2019MultiFC: A Real-World Multi-Domain Dataset for Evidence-Based Fact Checking of Claimscited by 169
  21. 2021Hurdles to Progress in Long-form Question Answeringcited by 166
  22. 2023Fundamental Limitations of Alignment in Large Language Modelscited by 152
  23. 2022GrIPS: Gradient-free, Edit-based Instruction Search for Prompting Large Language Modelscited by 127
  24. 2021Truthful AI: Developing and governing AI that does not liecited by 109
  25. 2021CREAK: A Dataset for Commonsense Reasoning over Entity Knowledgecited by 88
  26. 2021Improving Question Answering Model Robustness with Synthetic Adversarial Data Generationcited by 77
  27. 2022Gradient-Based Constrained Sampling from Language Modelscited by 64
  28. 2020Interactive Weak Supervision: Learning Useful Heuristics for Data Labelingcited by 52
  29. 2020Universal Adversarial Attacks with Natural Triggers for Text Classificationcited by 50
  30. 2022Toward Human Readable Prompt Tuning: Kubrick's The Shining is a good movie, and a good prompt too?cited by 49
  31. 2021Hard Choices in Artificial Intelligencecited by 26
  32. 2023Ground(less) Truth: A Causal Framework for Proxy Labels in Human-Algorithm Decision-Makingcited by 25
  33. 2022Humans are not Boltzmann Distributions: Challenges and Opportunities for Modelling Human Feedback and Interaction in Reinforcement Learningcited by 22
  34. 2020Quantifying Hypothesis Space Misspecification in Learning from Human-Robot Demonstrations and Physical Correctionscited by 19
  35. 2021Choice Set Misspecification in Reward Inferencecited by 13
  36. 2022Diversity Subsampling: Custom Subsamples from Large Data Setscited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 24; 2024: 40; 2025: 32; 2026: 15.

111 papers.

Sort:

By area:

  1. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,330
  2. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 743
  3. 2023The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasetscited by 673
  4. 2023A Comprehensive Overview of Large Language Modelscited by 639
  5. 2023Tree of Attacks: Jailbreaking Black-Box LLMs Automaticallycited by 619
  6. 2023Baseline Defenses for Adversarial Attacks Against Aligned Language Modelscited by 608
  7. 2023Catastrophic Jailbreak of Open-source LLMs via Exploiting Generationcited by 439
  8. 2023Jailbreak and Guard Aligned Language Models with Only Few In-Context Demonstrationscited by 401
  9. 2023Challenges and Applications of Large Language Modelscited by 251
  10. 2024Jailbreak Attacks and Defenses Against Large Language Models: A Surveycited by 215
  11. 2023Scalable and Transferable Black-Box Jailbreaks for Language Models via Persona Modulationcited by 202
  12. 2023AI Deception: A Survey of Examples, Risks, and Potential Solutionscited by 185
38 more
  1. 2023Can LLM-Generated Misinformation Be Detected?cited by 173
  2. 2023Sociotechnical Safety Evaluation of Generative AI Systemscited by 154
  3. 2024LLM Defenses Are Not Robust to Multi-Turn Human Jailbreaks Yetcited by 145
  4. 2024Attacks, Defenses and Evaluations for LLM Conversation Safety: A Surveycited by 124
  5. 2024Black-Box Access is Insufficient for Rigorous AI Auditscited by 123
  6. 2024Best Practices and Lessons Learned on Synthetic Datacited by 104
  7. 2024Coercing LLMs to do and reveal (almost) anythingcited by 88
  8. 2024Curiosity-driven Red-teaming for Large Language Modelscited by 83
  9. 2023FLIRT: Feedback Loop In-context Red Teamingcited by 81
  10. 2024Red-Teaming for Generative AI: Silver Bullet or Security Theater?cited by 81
  11. 2024LLMs know their vulnerabilities: Uncover Safety Gaps through Natural Distribution Shiftscited by 80
  12. 2024Truth is Universal: Robust Detection of Lies in LLMscited by 78
  13. 2024The Multilingual Alignment Prism: Aligning Global and Local Preferences to Reduce Harmcited by 67
  14. 2024Risk Taxonomy, Mitigation, and Assessment Benchmarks of Large Language Model Systemscited by 63
  15. 2023Can Language Models be Instructed to Protect Personal Information?cited by 46
  16. 2024Breaking Down the Defenses: A Comparative Survey of Attacks on Large Language Modelscited by 42
  17. 2023Hijacking Large Language Models via Adversarial In-Context Learningcited by 41
  18. 2024Learning to Generate Explainable Stock Predictions using Self-Reflective Large Language Modelscited by 35
  19. 2024Gradient-Based Language Model Red Teamingcited by 33
  20. 2024Crabs: Consuming Resource via Auto-generation for LLM-DoS Attack under Black-box Settingscited by 33
  21. 2025Chasing Moving Targets with Online Self-Play Reinforcement Learning for Safer Language Modelscited by 33
  22. 2024NetSafe: Exploring the Topological Safety of Multi-agent Networkscited by 29
  23. 2023Exploiting Novel GPT-4 APIscited by 27
  24. 2023Disentangling Perceptions of Offensiveness: Cultural and Moral Correlatescited by 26
  25. 2024Holistic Automated Red Teaming for Large Language Models through Top-Down Test Case Generation and Multi-turn Interactioncited by 25
  26. 2024Fundamental Problems With Model Editing: How Should Rational Belief Revision Work in LLMs?cited by 24
  27. 2024Diverse and Effective Red Teaming with Auto-generated Rewards and Multi-step Reinforcement Learningcited by 23
  28. 2025RAG LLMs are Not Safer: A Safety Analysis of Retrieval-Augmented Generation for Large Language Modelscited by 23
  29. 2023Language Model Unalignment: Parametric Red-Teaming to Expose Hidden Harms and Biasescited by 21
  30. 2024Mission Impossible: A Statistical Perspective on Jailbreaking LLMscited by 21
  31. 2025FORTRESS: Frontier Risk Evaluation for National Security and Public Safetycited by 20
  32. 2023Evolving Diverse Red-team Language Models in Multi-round Multi-agent Gamescited by 17
  33. 2024Improved Generation of Adversarial Examples Against Safety-aligned LLMscited by 17
  34. 2026Comparison requires valid measurement: Rethinking attack success rate comparisons in AI red teamingcited by 15
  35. 2023Understanding Multi-Turn Toxic Behaviors in Open-Domain Chatbotscited by 14
  36. 2024MoGU: A Framework for Enhancing Safety of Open-Sourced LLMs While Preserving Their Usabilitycited by 14
  37. 2025Building Safe GenAI Applications: An End-to-End Overview of Red Teaming for Large Language Modelscited by 11
  38. 2025Single-pass Detection of Jailbreaking Input in Large Language Modelscited by 10

50 most cited of 111 All 111 (JSON)