# JADES: A Universal Framework for Jailbreak Assessment via Decompositional Scoring

arXiv: https://arxiv.org/abs/2508.20848 · HTML: https://mlsanity.com/paper/2508.20848/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Placed in Jailbreaking, one of 12 directions in LLM safety and security.
- Builds on 29 of its 58 references that are in the map.
- First posted on arXiv 28 Aug 2025.

## Placed in

- LLM safety and security / [Jailbreaking](https://mlsanity.com/safety/jailbreaking/)

## Builds on (most cited first, 29 of 29)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Jailbroken: How Does LLM Safety Training Fail?](https://mlsanity.com/paper/2307.02483/) (2023)
- [HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal](https://mlsanity.com/paper/2402.04249/) (2024)
- [Jailbreaking Black Box Large Language Models in Twenty Queries](https://mlsanity.com/paper/2310.08419/) (2023)
- [A General Language Assistant as a Laboratory for Alignment](https://mlsanity.com/paper/2112.00861/) (2021)
- [Red Teaming Language Models with Language Models](https://mlsanity.com/paper/2202.03286/) (2022)
- [Tree of Attacks: Jailbreaking Black-Box LLMs Automatically](https://mlsanity.com/paper/2312.02119/) (2023)
- [GPTFUZZER: Red Teaming Large Language Models with Auto-Generated Jailbreak Prompts](https://mlsanity.com/paper/2309.10253/) (2023)
- [JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Models](https://mlsanity.com/paper/2404.01318/) (2024)
- ["Do Anything Now": Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Models](https://mlsanity.com/paper/2308.03825/) (2023)
- [Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attacks](https://mlsanity.com/paper/2404.02151/) (2024)
- [Catastrophic Jailbreak of Open-source LLMs via Exploiting Generation](https://mlsanity.com/paper/2310.06987/) (2023)
- [Detecting Language Model Attacks with Perplexity](https://mlsanity.com/paper/2308.14132/) (2023)
- [Low-Resource Languages Jailbreak GPT-4](https://mlsanity.com/paper/2310.02446/) (2023)
- [Exploiting Programmatic Behavior of LLMs: Dual-Use Through Standard Security Attacks](https://mlsanity.com/paper/2302.05733/) (2023)
- [Unsafe Diffusion: On the Generation of Unsafe Images and Hateful Memes From Text-To-Image Models](https://mlsanity.com/paper/2305.13873/) (2023)
- [COLD-Attack: Jailbreaking LLMs with Stealthiness and Controllability](https://mlsanity.com/paper/2402.08679/) (2024)
- [Fundamental Limitations of Alignment in Large Language Models](https://mlsanity.com/paper/2304.11082/) (2023)
- [Long-form factuality in large language models](https://mlsanity.com/paper/2403.18802/) (2024)
- [JailbreakRadar: Comprehensive Assessment of Jailbreak Attacks Against LLMs](https://mlsanity.com/paper/2402.05668/) (2024)
- [Tricking LLMs into Disobedience: Formalizing, Analyzing, and Detecting Jailbreaks](https://mlsanity.com/paper/2305.14965/) (2023)
- [SaLoRA: Safety-Alignment Preserved Low-Rank Adaptation](https://mlsanity.com/paper/2501.01765/) (2025)
- [Generated Data with Fake Privacy: Hidden Dangers of Fine-tuning Large Language Models on Generated Data](https://mlsanity.com/paper/2409.11423/) (2024)
- [Are Smarter LLMs Safer? Exploring Safety-Reasoning Trade-offs in Prompting and Fine-Tuning](https://mlsanity.com/paper/2502.09673/) (2025)
- [Excessive Reasoning Attack on Reasoning LLMs](https://mlsanity.com/paper/2506.14374/) (2025)
- [Reconstruct Your Previous Conversations! Comprehensively Investigating Privacy Leakage Risks in Conversations with GPT Models](https://mlsanity.com/paper/2402.02987/) (2024)
- [SOS! Soft Prompt Attack Against Open-Source Large Language Models](https://mlsanity.com/paper/2407.03160/) (2024)
- [FactLens: Benchmarking Fine-Grained Fact Verification](https://mlsanity.com/paper/2411.05980/) (2024)

## Built on by (most cited first, 14 of 14)

- [Adjacent Words, Divergent Intents: Jailbreaking Large Language Models via Task Concurrency](https://mlsanity.com/paper/2510.21189/) (2025)
- [Benchmark of Benchmarks: Unpacking Influence and Code Repository Quality in LLM Safety Benchmarks](https://mlsanity.com/paper/2603.04459/) (2026)
- [How Real is Your Jailbreak? Fine-grained Jailbreak Evaluation with Anchored Reference](https://mlsanity.com/paper/2601.03288/) (2026)
- [Scaling Patterns in Adversarial Alignment: Evidence from Multi-LLM Jailbreak Experiments](https://mlsanity.com/paper/2511.13788/) (2025)
- [Jailbreaking Large Language Models through Iterative Tool-Disguised Attacks via Reinforcement Learning](https://mlsanity.com/paper/2601.05466/) (2026)
- [When Understanding Becomes a Risk: Authenticity and Safety Risks in the Emerging Image Generation Paradigm](https://mlsanity.com/paper/2603.24079/) (2026)
- [Finding and Reactivating Post-Trained LLMs' Hidden Safety Mechanisms](https://mlsanity.com/paper/2604.00012/) (2026)
- [TrustLDM: Benchmarking Trustworthiness in Language Diffusion Models](https://mlsanity.com/paper/2606.00023/) (2026)
- [JailMeter: An Evidence-Based Evaluation Framework for Jailbreak Attacks on Large Language Models](https://mlsanity.com/paper/2607.19424/) (2026)
- [Risk-Adjusted Harm Scoring for Automated Red Teaming for LLMs in Financial Services](https://mlsanity.com/paper/2603.10807/) (2026)
- [Innocent Panels, Hateful Stories: Evaluating and Detecting Hateful Intent in Multi-Turn Visual Story Generation](https://mlsanity.com/paper/2608.05210/) (2026)
- [Validity-Aware Jailbreak Evaluation for Large Language Models](https://mlsanity.com/paper/2609.00498/) (2026)
- [An Empirical Measurement of Jailbreaking Evaluators](https://mlsanity.com/paper/2609.10594/) (2026)
- [Same Outcome, Different Evidence: Intent Recovery in LLM Safety Evaluation](https://mlsanity.com/paper/2610.11766/) (2026)

## Abstract (arXiv)

Accurately determining whether a jailbreak attempt has succeeded is a fundamental yet unresolved challenge. Existing evaluation methods rely on misaligned proxy indicators or naive holistic judgments. They frequently misinterpret model responses, leading to inconsistent and subjective assessments that misalign with human perception. To address this gap, we introduce JADES (Jailbreak Assessment via Decompositional Scoring), a universal jailbreak evaluation framework. Its key mechanism is to automatically decompose an input harmful question into a set of weighted sub-questions, score each sub-answer, and weight-aggregate the sub-scores into a final decision. JADES also incorporates an optional fact-checking module to strengthen the detection of hallucinations in jailbreak responses. We validate JADES on JailbreakQR, a newly introduced benchmark proposed in this work, consisting of 400 pairs of jailbreak prompts and responses, each meticulously annotated by humans. In a binary setting (success/failure), JADES achieves 98.5% agreement with human evaluators, outperforming strong baselines by over 9%. Re-evaluating five popular attacks on four LLMs reveals substantial overestimation (e.g., LAA's attack success rate on GPT-3.5-Turbo drops from 93% to 69%). Our results show that JADES could deliver accurate, consistent, and interpretable evaluations, providing a reliable basis for measuring future jailbreak attacks.