# [WIP] Jailbreak Paradox: The Achilles' Heel of LLMs

arXiv: https://arxiv.org/abs/2406.12702 · HTML: https://mlsanity.com/paper/2406.12702/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in Jailbreaking, one of 12 directions in LLM safety and security.
- Builds on 16 of its 23 references that are in the map.
- First posted on arXiv 18 Jun 2024.

## Placed in

- LLM safety and security / [Jailbreaking](https://mlsanity.com/safety/jailbreaking/)

## Builds on (most cited first, 16 of 16)

- [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021)
- [Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2204.05862/) (2022)
- [Universal and Transferable Adversarial Attacks on Aligned Language Models](https://mlsanity.com/paper/2307.15043/) (2023)
- [Jailbroken: How Does LLM Safety Training Fail?](https://mlsanity.com/paper/2307.02483/) (2023)
- [Beyond English-Centric Multilingual Machine Translation](https://mlsanity.com/paper/2010.11125/) (2020)
- [Can AI-Generated Text be Reliably Detected?](https://mlsanity.com/paper/2303.11156/) (2023)
- [MEGA: Multilingual Evaluation of Generative AI](https://mlsanity.com/paper/2303.12528/) (2023)
- [Inverse Scaling: When Bigger Isn't Better](https://mlsanity.com/paper/2306.09479/) (2023)
- [MasterKey: Automated Jailbreak Across Multiple Large Language Model Chatbots](https://mlsanity.com/paper/2307.08715/) (2023)
- [JailbreakRadar: Comprehensive Assessment of Jailbreak Attacks Against LLMs](https://mlsanity.com/paper/2402.05668/) (2024)
- [On the Computational Power of Transformers and its Implications in Sequence Modeling](https://mlsanity.com/paper/2006.09286/) (2020)
- [MEGAVERSE: Benchmarking Large Language Models Across Languages, Modalities, Models and Tasks](https://mlsanity.com/paper/2311.07463/) (2023)
- [Tricking LLMs into Disobedience: Formalizing, Analyzing, and Detecting Jailbreaks](https://mlsanity.com/paper/2305.14965/) (2023)
- [Tamil-Llama: A New Tamil Language Model Based on Llama 2](https://mlsanity.com/paper/2311.05845/) (2023)
- [Bergeron: Combating Adversarial Attacks through a Conscience-Based Alignment Framework](https://mlsanity.com/paper/2312.00029/) (2023)
- [Testing the Limits of Jailbreaking Defenses with the Purple Problem](https://mlsanity.com/paper/2403.14725/) (2024)

## Built on by (most cited first, 2 of 2)

- [CourtGuard: A Local, Multiagent Prompt Injection Classifier](https://mlsanity.com/paper/2510.19844/) (2025)
- [A Real-Time, Self-Tuning Moderator Framework for Adversarial Prompt Detection](https://mlsanity.com/paper/2508.07139/) (2025)

## Abstract (arXiv)

We introduce two paradoxes concerning jailbreak of foundation models: First, it is impossible to construct a perfect jailbreak classifier, and second, a weaker model cannot consistently detect whether a stronger (in a pareto-dominant sense) model is jailbroken or not. We provide formal proofs for these paradoxes and a short case study on Llama and GPT4-o to demonstrate this. We discuss broader theoretical and practical repercussions of these results.