mlsanityData to 14 Oct 2026

The Jailbreak Tax: How Useful are Your Jailbreak Outputs?

arXiv 2504.10694 ↗First posted 14 Apr 2025cs.LG cs.AI cs.CRcited by 28

On the map: LLM safety and security → Jailbreaking

Abstract (arXiv)

Jailbreak attacks bypass the guardrails of large language models to produce harmful outputs. In this paper, we ask whether the model outputs produced by existing jailbreaks are actually useful. For example, when jailbreaking a model to give instructions for building a bomb, does the jailbreak yield good instructions? Since the utility of most unsafe answers (e.g., bomb instructions) is hard to evaluate rigorously, we build new jailbreak evaluation sets with known ground truth answers, by aligning models to refuse questions related to benign and easy-to-evaluate topics (e.g., biology or math). Our evaluation of eight representative jailbreaks across five utility benchmarks reveals a consistent drop in model utility in jailbroken responses, which we term the jailbreak tax. For example, while all jailbreaks we tested bypass guardrails in models aligned to refuse to answer math, this comes at the expense of a drop of up to 92% in accuracy. Overall, our work proposes the jailbreak tax as a new important metric in AI safety, and introduces benchmarks to evaluate existing and future jailbreaks. We make the benchmark available at https://github.com/ethz-spylab/jailbreak-tax

Key facts. Cited by 28 papers in the map (data through 14 Oct 2026). Placed in Jailbreaking, one of 12 directions in LLM safety and security. Builds on 19 of its 22 references that are in the map. First posted on arXiv 14 Apr 2025.

Builds on

19 of 22 references.

Closest ancestors · 10

Same direction, newest first

  1. 2024Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attacks2 Apr
  2. 2024JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Models28 Mar
  3. 2024A StrongREJECT for Empty Jailbreaks15 Feb
  4. 2023Tree of Attacks: Jailbreaking Black-Box LLMs Automatically4 Dec
  5. 2023Multilingual Jailbreak Challenges in Large Language Models10 Oct
  6. 2023AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Models3 Oct
  7. 2023Low-Resource Languages Jailbreak GPT-43 Oct
  8. 2023GPTFUZZER: Red Teaming Large Language Models with Auto-Generated Jailbreak Prompts19 Sep
  9. 2023Universal and Transferable Adversarial Attacks on Aligned Language Models27 Jul
  10. 2023Jailbroken: How Does LLM Safety Training Fail?5 Jul

Other references · 9

  1. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  3. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  4. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  5. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,429
  6. 2024The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearningcited by 509
  7. 2024AgentHarm: A Benchmark for Measuring Harmfulness of LLM Agentscited by 437
  8. 2024Assessing the Brittleness of Safety Alignment via Pruning and Low-Rank Modificationscited by 231
  9. 2024On the Societal Impact of Open Foundation Modelscited by 65

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 12; 2026: 16.

28 papers.

Sort:

By area:

  1. 2026Constitutional Classifiers++: Efficient Production-Grade Defenses against Universal Jailbreakscited by 41
  2. 2025ReliabilityRAG: Effective and Provably Robust Defense for RAG-based Web-Searchcited by 18
  3. 2026Learning to Inject: Automated Prompt Injection via Reinforcement Learningcited by 16
  4. 2025Jailbreak-Tuning: Models Efficiently Learn Jailbreak Susceptibilitycited by 9
  5. 2025Benchmarking Misuse Mitigation Against Covert Adversariescited by 8
  6. 2025All Code, No Thought: Current Language Models Struggle to Reason in Ciphered Languagecited by 8
  7. 2025Bypassing Prompt Guards in Production with Controlled-Release Promptingcited by 7
  8. 2025Jailbreak Transferability Emerges from Shared Representationscited by 6
  9. 2025Learning the Wrong Lessons: Syntactic-Domain Spurious Correlations in Language Modelscited by 6
  10. 2026JailNewsBench: Multi-Lingual and Regional Benchmark for Fake News Generation under Jailbreak Attackscited by 6
  11. 2025Idea2Plan: Exploring AI-Powered Research Planningcited by 4
  12. 2026Jailbreak Scaling Laws for Large Language Models: Polynomial-Exponential Crossovercited by 4
16 more
  1. 2025Beyond Jailbreaks: Revealing Stealthier and Broader LLM Security Risks Stemming from Alignment Failurescited by 3
  2. 2025Jailbreaking Large Language Models Through Content Concretizationcited by 2
  3. 2025Multi-Agent Framework for Threat Mitigation and Resilience in AI-Based Systemscited by 2
  4. 2025bi-GRPO: Bidirectional Optimization for Jailbreak Backdoor Injection on LLMscited by 1
  5. 2026Syntax- and Compilation-Preserving Evasion of LLM Vulnerability Detectorscited by 1
  6. 2026Jailbroken Frontier Models Retain Their Capabilitiescited by 1
  7. 2026AI Security Leaderboard: Methodology, Results and Minimal Standardcited by 1
  8. 2026HarmProfile: Characterizing Harmful Distributions in Frontier LLMscited by 1
  9. 2026Jailbreaking LLMs via Calibrationcited by 0
  10. 2026ThaiSafetyBench: Assessing Language Model Safety in Thai Cultural Contextscited by 0
  11. 2026AI Security Policy Should Assess Systems, Not Only Modelscited by 0
  12. 2026Greedy Coordinate Diffusion: Effective and Semantically Coherent Adversarial Attacks via Diffusion Guidancecited by 0
  13. 2026NeuronFuzz: Safety Neuron Guided Fuzzing for LLM Safety Evaluationcited by 0
  14. 2026Validity-Aware Jailbreak Evaluation for Large Language Modelscited by 0
  15. 2026VEX-Bench: Benchmarking Verification Complexity of LLM-Generated Misinformationcited by 0
  16. 2026MLLMs Fail to Refuse when Using Tools Agenticallycited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Rethinking How to Evaluate Language Model Jailbreak
  2. 2024JailbreakEval: An Integrated Toolkit for Evaluating Jailbreak Attempts Against Large Language Models
  3. 2024"Not Aligned" is Not "Malicious": Being Careful about Hallucinations of Large Language Models' Jailbreak
  4. 2026JailMeter: An Evidence-Based Evaluation Framework for Jailbreak Attacks on Large Language Models
  5. 2026An Empirical Measurement of Jailbreaking Evaluators
  6. 2026How Jailbreak Attacks Inform Safety Alignment: A Defender-Centric, Shapley-Based Evaluation of Jailbreak Contributions
  7. 2024AttackEval: How to Evaluate the Effectiveness of Jailbreak Attacking on Large Language Models
  8. 2026How Real is Your Jailbreak? Fine-grained Jailbreak Evaluation with Anchored Reference