mlsanityData to 7 Oct 2026

A StrongREJECT for Empty Jailbreaks

arXiv 2402.10260 ↗First posted 15 Feb 2024cs.LG cs.CL cs.CRcited by 352

On the map: LLM safety and security → Jailbreaking

Abstract (arXiv)

Most jailbreak papers claim the jailbreaks they propose are highly effective, often boasting near-100% attack success rates. However, it is perhaps more common than not for jailbreak developers to substantially exaggerate the effectiveness of their jailbreaks. We suggest this problem arises because jailbreak researchers lack a standard, high-quality benchmark for evaluating jailbreak performance, leaving researchers to create their own. To create a benchmark, researchers must choose a dataset of forbidden prompts to which a victim model will respond, along with an evaluation method that scores the harmfulness of the victim model's responses. We show that existing benchmarks suffer from significant shortcomings and introduce the StrongREJECT benchmark to address these issues. StrongREJECT's dataset contains prompts that victim models must answer with specific, harmful information, while its automated evaluator measures the extent to which a response gives useful information to forbidden prompts. In doing so, the StrongREJECT evaluator achieves state-of-the-art agreement with human judgments of jailbreak effectiveness. Notably, we find that existing evaluation methods significantly overstate jailbreak effectiveness compared to human judgments and the StrongREJECT evaluator. We describe a surprising and novel phenomenon that explains this discrepancy: jailbreaks bypassing a victim model's safety fine-tuning tend to reduce its capabilities. Together, our findings underscore the need for researchers to use a high-quality benchmark, such as StrongREJECT, when developing new jailbreak attacks. We release the StrongREJECT code and data at https://strong-reject.readthedocs.io/en/latest/.

Key facts. Cited by 352 papers in the map (data through 7 Oct 2026). Placed in Jailbreaking, one of 12 directions in LLM safety and security. Builds on 31 of its 64 references that are in the map. First posted on arXiv 15 Feb 2024.

Builds on

31 of 64 references.

Closest ancestors · 16

Same direction, newest first

  1. 2024How Johnny Can Persuade LLMs to Jailbreak Them: Rethinking Persuasion to Challenge AI Safety by Humanizing LLMs12 Jan
  2. 2023Tree of Attacks: Jailbreaking Black-Box LLMs Automatically4 Dec
  3. 2023Cognitive Overload: Jailbreaking Large Language Models with Overloaded Logical Thinking16 Nov
  4. 2023Scalable and Transferable Black-Box Jailbreaks for Language Models via Persona Modulation6 Nov
  5. 2023DeepInception: Hypnotize Large Language Model to Be Jailbreaker6 Nov
  6. 2023Jailbreaking Black Box Large Language Models in Twenty Queries12 Oct
  7. 2023Catastrophic Jailbreak of Open-source LLMs via Exploiting Generation10 Oct
  8. 2023SmoothLLM: Defending Large Language Models Against Jailbreaking Attacks5 Oct
  9. 2023Low-Resource Languages Jailbreak GPT-43 Oct
  10. 2023GPTFUZZER: Red Teaming Large Language Models with Auto-Generated Jailbreak Prompts19 Sep
  11. 2023Open Sesame! Universal Black Box Jailbreaking of Large Language Models4 Sep
  12. 2023"Do Anything Now": Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Models7 Aug
4 more
  1. 2023Universal and Transferable Adversarial Attacks on Aligned Language Models27 Jul
  2. 2023MasterKey: Automated Jailbreak Across Multiple Large Language Model Chatbots16 Jul
  3. 2023Jailbroken: How Does LLM Safety Training Fail?5 Jul
  4. 2023Jailbreaking ChatGPT via Prompt Engineering: An Empirical Study23 May

Other references · 15

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  3. 2023Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injectioncited by 1,339
  4. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,330
  5. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!cited by 1,106
  6. 2022Red Teaming Language Models with Language Modelscited by 983
  7. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learnedcited by 702
  8. 2023Exploiting Programmatic Behavior of LLMs: Dual-Use Through Standard Security Attackscited by 268
  9. 2022On Second Thought, Let's Not Think Step by Step! Bias and Toxicity in Zero-Shot Reasoningcited by 191
  10. 2023Image Hijacks: Adversarial Images can Control Generative Models at Runtimecited by 165
  11. 2023Removing RLHF Protections in GPT-4 via Fine-Tuningcited by 165
  12. 2022Why Should Adversarial Perturbations be Imperceptible? Rethink the Research Paradigm in Adversarial NLPcited by 99
3 more
  1. 2023Use of LLMs for Illicit Purposes: Threats, Prevention Measures, and Vulnerabilitiescited by 76
  2. 2023Goal-Oriented Prompt Attack and Safety Evaluation for LLMscited by 23
  3. 2023LoFT: Local Proxy Fine-tuning For Improving Transferability Of Adversarial Attacks Against Large Language Modelcited by 21

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 32; 2025: 134; 2026: 186.

352 papers.

Sort:

By area:

  1. 2025gpt-oss-120b & gpt-oss-20b Model Cardcited by 1,258
  2. 2024OpenAI o1 System Cardcited by 1,220
  3. 2024Refusal in Language Models Is Mediated by a Single Directioncited by 960
  4. 2024JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Modelscited by 494
  5. 2024Deliberative Alignment: Reasoning Enables Safer Language Modelscited by 292
  6. 2024SORRY-Bench: Systematically Evaluating Large Language Model Safety Refusalcited by 224
  7. 2024Jailbreak Attacks and Defenses Against Large Language Models: A Surveycited by 215
  8. 2024Latent Adversarial Training Improves Robustness to Persistent Harmful Behaviors in LLMscited by 158
  9. 2025Emergent Misalignment: Narrow finetuning can produce broadly misaligned LLMscited by 158
  10. 2024AdvPrompter: Fast Adaptive Adversarial Prompting for LLMscited by 146
  11. 2025SafeChain: Safety of Language Models with Long Chain-of-Thought Reasoning Capabilitiescited by 133
  12. 2024Know Your Limits: A Survey of Abstention in Large Language Modelscited by 116
38 more
  1. 2025A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deploymentcited by 110
  2. 2025The Geometry of Refusal in Large Language Models: Concept Cones and Representational Independencecited by 104
  3. 2024SafetyPrompts: a Systematic Review of Open Datasets for Evaluating and Improving Large Language Model Safetycited by 75
  4. 2024FlipAttack: Jailbreak LLMs via Flippingcited by 73
  5. 2025STAIR: Improving Safety Alignment with Introspective Reasoningcited by 72
  6. 2024Best-of-N Jailbreakingcited by 71
  7. 2024JailbreakZoo: Survey, Landscapes, and Horizons in Jailbreaking Large Language and Vision-Language Modelscited by 67
  8. 2025STAR-1: Safer Alignment of Reasoning LLMs with 1K Datacited by 67
  9. 2025Poisoning Attacks on LLMs Require a Near-constant Number of Poison Samplescited by 66
  10. 2024Uncovering Safety Risks of Large Language Models through Concept Activation Vectorcited by 63
  11. 2024Introducing v0.5 of the AI Safety Benchmark from MLCommonscited by 60
  12. 2024Robust LLM safeguarding via refusal feature adversarial trainingcited by 60
  13. 2025Trading Inference-Time Compute for Adversarial Robustnesscited by 60
  14. 2025RealSafe-R1: Safety-Aligned DeepSeek-R1 without Compromising Reasoning Capabilitycited by 58
  15. 2025FalseReject: A Resource for Improving Contextual Safety and Mitigating Over-Refusals in LLMs via Structured Reasoningcited by 53
  16. 2024Limits to scalable evaluation at the frontier: LLM as Judge won't beat twice the datacited by 46
  17. 2025Angular Steering: Behavior Control via Rotation in Activation Spacecited by 45
  18. 2024R2-Guard: Robust Reasoning Enabled LLM Guardrail via Knowledge-Enhanced Logical Reasoningcited by 41
  19. 2024Automated Red Teaming with GOAT: the Generative Offensive Agent Testercited by 41
  20. 2025The Hidden Dimensions of LLM Alignment: A Multi-Dimensional Analysis of Orthogonal Safety Directionscited by 38
  21. 2025Model Tampering Attacks Enable More Rigorous Evaluations of LLM Capabilitiescited by 36
  22. 2026Constitutional Classifiers++: Efficient Production-Grade Defenses against Universal Jailbreakscited by 36
  23. 2024InfAlign: Inference-aware language model alignmentcited by 34
  24. 2025Chasing Moving Targets with Online Self-Play Reinforcement Learning for Safer Language Modelscited by 33
  25. 2025The Devil behind the mask: An emergent safety vulnerability of Diffusion LLMscited by 32
  26. 2025Inoculation Prompting: Instructing LLMs to misbehave at train-time improves test-time alignmentcited by 32
  27. 2024Backtracking Improves Generation Safetycited by 31
  28. 2024JailbreakEval: An Integrated Toolkit for Evaluating Jailbreak Attempts Against Large Language Modelscited by 28
  29. 2025Adversarial Reasoning at Jailbreaking Timecited by 27
  30. 2024Mind the Inconspicuous: Revealing the Hidden Weakness in Aligned LLMs' Refusal Boundariescited by 26
  31. 2025The Jailbreak Tax: How Useful are Your Jailbreak Outputs?cited by 26
  32. 2024AdvPrefix: An Objective for Nuanced LLM Jailbreakscited by 25
  33. 2025Multilingual and Multi-Accent Jailbreaking of Audio LLMscited by 25
  34. 2025Mitigating Deceptive Alignment via Self-Monitoringcited by 25
  35. 2024What Features in Prompts Jailbreak LLMs? Investigating the Mechanisms Behind Attackscited by 24
  36. 2025o3-mini vs DeepSeek-R1: Which One is Safer?cited by 24
  37. 2025SAFEPATH: Preventing Harmful Reasoning in Chain-of-Thought via Early Alignmentcited by 24
  38. 2025The Scales of Justitia: A Comprehensive Survey on Safety Evaluation of LLMscited by 24

50 most cited of 352 All 352 (JSON)