mlsanityData to 7 Oct 2026

HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal

arXiv 2402.04249 ↗First posted 6 Feb 2024cs.LG cs.AI cs.CLcited by 1,330

On the map: LLM safety and security → Prompt injection and agent security

Abstract (arXiv)

Automated red teaming holds substantial promise for uncovering and mitigating the risks associated with the malicious use of large language models (LLMs), yet the field lacks a standardized evaluation framework to rigorously assess new methods. To address this issue, we introduce HarmBench, a standardized evaluation framework for automated red teaming. We identify several desirable properties previously unaccounted for in red teaming evaluations and systematically design HarmBench to meet these criteria. Using HarmBench, we conduct a large-scale comparison of 18 red teaming methods and 33 target LLMs and defenses, yielding novel insights. We also introduce a highly efficient adversarial training method that greatly enhances LLM robustness across a wide range of attacks, demonstrating how HarmBench enables codevelopment of attacks and defenses. We open source HarmBench at https://github.com/centerforaisafety/HarmBench.

Key facts. Cited by 1,330 papers in the map (data through 7 Oct 2026). Placed in Prompt injection and agent security, one of 12 directions in LLM safety and security. Builds on 75 of its 107 references that are in the map. First posted on arXiv 6 Feb 2024.

Builds on

75 of 107 references.

Closest ancestors · 4

Same direction, newest first

  1. 2023MART: Improving LLM Safety with Multi-round Automatic Red-Teaming13 Nov
  2. 2023Explore, Establish, Exploit: Red Teaming Language Models from Scratch15 Jun
  3. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned23 Aug
  4. 2022Red Teaming Language Models with Language Models7 Feb

Other references · 71

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  6. 2023Visual Instruction Tuningcited by 7,792
  7. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
  8. 2013Intriguing properties of neural networkscited by 5,114
  9. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  10. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  11. 2023Qwen Technical Reportcited by 3,273
  12. 2023Mistral 7Bcited by 3,065
38 more
  1. 2016Towards Evaluating the Robustness of Neural Networkscited by 2,976
  2. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  3. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
  4. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  5. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,565
  6. 2020Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attackscited by 1,442
  7. 2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,326
  8. 2019Certified Adversarial Robustness via Randomized Smoothingcited by 1,324
  9. 2018Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examplescited by 1,259
  10. 2023Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversationscited by 1,124
  11. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!cited by 1,106
  12. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 825
  13. 2023AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Modelscited by 738
  14. 2019Universal Adversarial Triggers for Attacking and Analyzing NLPcited by 709
  15. 2023Baichuan 2: Open Large-scale Language Modelscited by 664
  16. 2019Is BERT Really Robust? A Strong Baseline for Natural Language Attack on Text Classification and Entailmentcited by 663
  17. 2020RobustBench: a standardized adversarial robustness benchmarkcited by 659
  18. 2019Adversarial Training for Free!cited by 638
  19. 2023Tree of Attacks: Jailbreaking Black-Box LLMs Automaticallycited by 619
  20. 2023Baseline Defenses for Adversarial Attacks Against Aligned Language Modelscited by 608
  21. 2024How Johnny Can Persuade LLMs to Jailbreak Them: Rethinking Persuasion to Challenge AI Safety by Humanizing LLMscited by 533
  22. 2017Adversarial Patchcited by 514
  23. 2019Using Self-Supervised Learning Can Improve Model Robustness and Uncertaintycited by 511
  24. 2023Zephyr: Direct Distillation of LM Alignmentcited by 498
  25. 2020BERT-ATTACK: Adversarial Attack Against BERT Using BERTcited by 488
  26. 2023Jailbreaking ChatGPT via Prompt Engineering: An Empirical Studycited by 478
  27. 2023"Do Anything Now": Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Modelscited by 457
  28. 2023Catastrophic Jailbreak of Open-source LLMs via Exploiting Generationcited by 439
  29. 2019Unlabeled Data Improves Adversarial Robustnesscited by 415
  30. 2018TextBugger: Generating Adversarial Text Against Real-world Applicationscited by 376
  31. 2019Using Pre-Training Can Improve Model Robustness and Uncertaintycited by 373
  32. 2023NeMo Guardrails: A Toolkit for Controllable and Safe LLM Applications with Programmable Railscited by 347
  33. 2022A Holistic Approach to Undesired Content Detection in the Real Worldcited by 337
  34. 2023Hard Prompts Made Easy: Gradient-Based Discrete Optimization for Prompt Tuning and Discoverycited by 329
  35. 2023Visual Adversarial Examples Jailbreak Aligned Large Language Modelscited by 307
  36. 2023Are aligned neural networks adversarially aligned?cited by 305
  37. 2018Adversarial Example Generation with Syntactically Controlled Paraphrase Networkscited by 282
  38. 2019FreeLB: Enhanced Adversarial Training for Natural Language Understandingcited by 261

50 most cited of 71 All 71 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 192; 2025: 458; 2026: 680.

1,330 papers.

Sort:

By area:

  1. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Trainingcited by 988
  2. 2024Refusal in Language Models Is Mediated by a Single Directioncited by 960
  3. 2024JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Modelscited by 494
  4. 2024The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearningcited by 488
  5. 2024Sora: A Review on Background, Technology, Limitations, and Opportunities of Large Vision Modelscited by 467
  6. 2024WildGuard: Open One-Stop Moderation Tools for Safety Risks, Jailbreaks, and Refusals of LLMscited by 460
  7. 2024Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attackscited by 450
  8. 2024AgentHarm: A Benchmark for Measuring Harmfulness of LLM Agentscited by 406
  9. 2024Great, Now Write an Article About That: The Crescendo Multi-Turn LLM Jailbreak Attackcited by 364
  10. 2024A StrongREJECT for Empty Jailbreakscited by 352
  11. 2024Alignment faking in large language modelscited by 316
  12. 2024Improving Alignment and Robustness with Circuit Breakerscited by 302
38 more
  1. 2025OpenThoughts: Data Recipes for Reasoning Modelscited by 287
  2. 2024Agent-SafetyBench: Evaluating the Safety of LLM Agentscited by 280
  3. 2024SALAD-Bench: A Hierarchical and Comprehensive Safety Benchmark for Large Language Modelscited by 225
  4. 2024SORRY-Bench: Systematically Evaluating Large Language Model Safety Refusalcited by 224
  5. 2024ShieldGemma: Generative AI Content Moderation Based on Gemmacited by 224
  6. 2024Jailbreak Attacks and Defenses Against Large Language Models: A Surveycited by 215
  7. 2024AutoDAN-Turbo: A Lifelong Agent for Strategy Self-Exploration to Jailbreak LLMscited by 185
  8. 2024COLD-Attack: Jailbreaking LLMs with Stealthiness and Controllabilitycited by 162
  9. 2024AmpleGCG: Learning a Universal and Transferable Generative Model of Adversarial Suffixes for Jailbreaking Both Open and Closed LLMscited by 160
  10. 2024Latent Adversarial Training Improves Robustness to Persistent Harmful Behaviors in LLMscited by 158
  11. 2024AdvPrompter: Fast Adaptive Adversarial Prompting for LLMscited by 146
  12. 2024LLM Defenses Are Not Robust to Multi-Turn Human Jailbreaks Yetcited by 145
  13. 2025SafeChain: Safety of Language Models with Long Chain-of-Thought Reasoning Capabilitiescited by 133
  14. 2024Tamper-Resistant Safeguards for Open-Weight LLMscited by 132
  15. 2024AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agentscited by 129
  16. 2025AbstentionBench: Reasoning LLMs Fail on Unanswerable Questionscited by 129
  17. 2025Qwen3Guard Technical Reportcited by 125
  18. 2025Aegis2.0: A Diverse AI Safety Dataset and Risks Taxonomy for Alignment of LLM Guardrailscited by 122
  19. 2024Know Your Limits: A Survey of Abstention in Large Language Modelscited by 116
  20. 2025Adaptive Attacks Break Defenses Against Indirect Prompt Injection Attacks on LLM Agentscited by 112
  21. 2025X-Teaming: Multi-Turn Jailbreaks and Defenses with Adaptive Multi-Agentscited by 112
  22. 2025Natural Emergent Misalignment from Reward Hacking in Production RLcited by 111
  23. 2025A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deploymentcited by 110
  24. 2025The Attacker Moves Second: Stronger Adaptive Attacks Bypass Defenses Against Llm Jailbreaks and Prompt Injectionscited by 108
  25. 2024Efficient Adversarial Training in LLMs with Continuous Attackscited by 104
  26. 2025OS-Harm: A Benchmark for Measuring Safety of Computer Use Agentscited by 104
  27. 2024Improved Techniques for Optimization-Based Jailbreaking on Large Language Modelscited by 98
  28. 2025SafeArena: Evaluating the Safety of Autonomous Web Agentscited by 97
  29. 2024A Comprehensive Study of Jailbreak Attack versus Defense for Large Language Modelscited by 95
  30. 2024Making Them Ask and Answer: Jailbreaking Large Language Models in Few Queries via Disguise and Reconstructioncited by 95
  31. 2024Representation Noising: A Defence Mechanism Against Harmful Finetuningcited by 93
  32. 2024Attacking Large Language Models with Projected Gradient Descentcited by 91
  33. 2024Steering Language Model Refusal with Sparse Autoencoderscited by 89
  34. 2025GuardReasoner: Towards Reasoning-based LLM Safeguardscited by 89
  35. 2024Soft Prompt Threats: Attacking Safety Alignment and Unlearning in Open-Source LLMs through the Embedding Spacecited by 88
  36. 2024RedCode: Risky Code Execution and Generation Benchmark for Code Agentscited by 87
  37. 2024Harmful Fine-tuning Attacks and Defenses for Large Language Models: A Surveycited by 86
  38. 2024LLMs know their vulnerabilities: Uncover Safety Gaps through Natural Distribution Shiftscited by 80

50 most cited of 1,330 All 1,330 (JSON)