# The Jailbreak Tax: How Useful are Your Jailbreak Outputs?

arXiv: https://arxiv.org/abs/2504.10694 · HTML: https://mlsanity.com/paper/2504.10694/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- Placed in Jailbreaking, one of 12 directions in LLM safety and security.
- Builds on 19 of its 22 references that are in the map.
- First posted on arXiv 14 Apr 2025.

## Placed in

- LLM safety and security / [Jailbreaking](https://mlsanity.com/safety/jailbreaking/)

## Builds on (most cited first, 19 of 19)

- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2204.05862/) (2022)
- [Universal and Transferable Adversarial Attacks on Aligned Language Models](https://mlsanity.com/paper/2307.15043/) (2023)
- [Jailbroken: How Does LLM Safety Training Fail?](https://mlsanity.com/paper/2307.02483/) (2023)
- [HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal](https://mlsanity.com/paper/2402.04249/) (2024)
- [AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Models](https://mlsanity.com/paper/2310.04451/) (2023)
- [Tree of Attacks: Jailbreaking Black-Box LLMs Automatically](https://mlsanity.com/paper/2312.02119/) (2023)
- [GPTFUZZER: Red Teaming Large Language Models with Auto-Generated Jailbreak Prompts](https://mlsanity.com/paper/2309.10253/) (2023)
- [JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Models](https://mlsanity.com/paper/2404.01318/) (2024)
- [The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearning](https://mlsanity.com/paper/2403.03218/) (2024)
- [Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attacks](https://mlsanity.com/paper/2404.02151/) (2024)
- [AgentHarm: A Benchmark for Measuring Harmfulness of LLM Agents](https://mlsanity.com/paper/2410.09024/) (2024)
- [A StrongREJECT for Empty Jailbreaks](https://mlsanity.com/paper/2402.10260/) (2024)
- [Low-Resource Languages Jailbreak GPT-4](https://mlsanity.com/paper/2310.02446/) (2023)
- [Multilingual Jailbreak Challenges in Large Language Models](https://mlsanity.com/paper/2310.06474/) (2023)
- [Assessing the Brittleness of Safety Alignment via Pruning and Low-Rank Modifications](https://mlsanity.com/paper/2402.05162/) (2024)
- [On the Societal Impact of Open Foundation Models](https://mlsanity.com/paper/2403.07918/) (2024)

## Built on by (most cited first, 28 of 28)

- [Constitutional Classifiers++: Efficient Production-Grade Defenses against Universal Jailbreaks](https://mlsanity.com/paper/2601.04603/) (2026)
- [ReliabilityRAG: Effective and Provably Robust Defense for RAG-based Web-Search](https://mlsanity.com/paper/2509.23519/) (2025)
- [Learning to Inject: Automated Prompt Injection via Reinforcement Learning](https://mlsanity.com/paper/2602.05746/) (2026)
- [Jailbreak-Tuning: Models Efficiently Learn Jailbreak Susceptibility](https://mlsanity.com/paper/2507.11630/) (2025)
- [Benchmarking Misuse Mitigation Against Covert Adversaries](https://mlsanity.com/paper/2506.06414/) (2025)
- [All Code, No Thought: Current Language Models Struggle to Reason in Ciphered Language](https://mlsanity.com/paper/2510.09714/) (2025)
- [Bypassing Prompt Guards in Production with Controlled-Release Prompting](https://mlsanity.com/paper/2510.01529/) (2025)
- [Jailbreak Transferability Emerges from Shared Representations](https://mlsanity.com/paper/2506.12913/) (2025)
- [Learning the Wrong Lessons: Syntactic-Domain Spurious Correlations in Language Models](https://mlsanity.com/paper/2509.21155/) (2025)
- [JailNewsBench: Multi-Lingual and Regional Benchmark for Fake News Generation under Jailbreak Attacks](https://mlsanity.com/paper/2603.01291/) (2026)
- [Idea2Plan: Exploring AI-Powered Research Planning](https://mlsanity.com/paper/2510.24891/) (2025)
- [Jailbreak Scaling Laws for Large Language Models: Polynomial-Exponential Crossover](https://mlsanity.com/paper/2603.11331/) (2026)
- [Beyond Jailbreaks: Revealing Stealthier and Broader LLM Security Risks Stemming from Alignment Failures](https://mlsanity.com/paper/2506.07402/) (2025)
- [Jailbreaking Large Language Models Through Content Concretization](https://mlsanity.com/paper/2509.12937/) (2025)
- [Multi-Agent Framework for Threat Mitigation and Resilience in AI-Based Systems](https://mlsanity.com/paper/2512.23132/) (2025)
- [bi-GRPO: Bidirectional Optimization for Jailbreak Backdoor Injection on LLMs](https://mlsanity.com/paper/2509.19775/) (2025)
- [Syntax- and Compilation-Preserving Evasion of LLM Vulnerability Detectors](https://mlsanity.com/paper/2602.00305/) (2026)
- [Jailbroken Frontier Models Retain Their Capabilities](https://mlsanity.com/paper/2605.00267/) (2026)
- [AI Security Leaderboard: Methodology, Results and Minimal Standard](https://mlsanity.com/paper/2608.03070/) (2026)
- [HarmProfile: Characterizing Harmful Distributions in Frontier LLMs](https://mlsanity.com/paper/2608.14577/) (2026)
- [Jailbreaking LLMs via Calibration](https://mlsanity.com/paper/2602.00619/) (2026)
- [ThaiSafetyBench: Assessing Language Model Safety in Thai Cultural Contexts](https://mlsanity.com/paper/2603.04992/) (2026)
- [AI Security Policy Should Assess Systems, Not Only Models](https://mlsanity.com/paper/2605.09504/) (2026)
- [Greedy Coordinate Diffusion: Effective and Semantically Coherent Adversarial Attacks via Diffusion Guidance](https://mlsanity.com/paper/2606.15531/) (2026)
- [NeuronFuzz: Safety Neuron Guided Fuzzing for LLM Safety Evaluation](https://mlsanity.com/paper/2608.26222/) (2026)
- [Validity-Aware Jailbreak Evaluation for Large Language Models](https://mlsanity.com/paper/2609.00498/) (2026)
- [VEX-Bench: Benchmarking Verification Complexity of LLM-Generated Misinformation](https://mlsanity.com/paper/2609.35028/) (2026)
- [MLLMs Fail to Refuse when Using Tools Agentically](https://mlsanity.com/paper/2610.03938/) (2026)

## Abstract (arXiv)

Jailbreak attacks bypass the guardrails of large language models to produce harmful outputs. In this paper, we ask whether the model outputs produced by existing jailbreaks are actually useful. For example, when jailbreaking a model to give instructions for building a bomb, does the jailbreak yield good instructions? Since the utility of most unsafe answers (e.g., bomb instructions) is hard to evaluate rigorously, we build new jailbreak evaluation sets with known ground truth answers, by aligning models to refuse questions related to benign and easy-to-evaluate topics (e.g., biology or math). Our evaluation of eight representative jailbreaks across five utility benchmarks reveals a consistent drop in model utility in jailbroken responses, which we term the jailbreak tax. For example, while all jailbreaks we tested bypass guardrails in models aligned to refuse to answer math, this comes at the expense of a drop of up to 92% in accuracy. Overall, our work proposes the jailbreak tax as a new important metric in AI safety, and introduces benchmarks to evaluate existing and future jailbreaks. We make the benchmark available at https://github.com/ethz-spylab/jailbreak-tax