mlsanityData to 7 Oct 2026

[WIP] Jailbreak Paradox: The Achilles' Heel of LLMs

arXiv 2406.12702 ↗First posted 18 Jun 2024cs.CLcited by 2

On the map: LLM safety and security → Jailbreaking

Abstract (arXiv)

We introduce two paradoxes concerning jailbreak of foundation models: First, it is impossible to construct a perfect jailbreak classifier, and second, a weaker model cannot consistently detect whether a stronger (in a pareto-dominant sense) model is jailbroken or not. We provide formal proofs for these paradoxes and a short case study on Llama and GPT4-o to demonstrate this. We discuss broader theoretical and practical repercussions of these results.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Jailbreaking, one of 12 directions in LLM safety and security. Builds on 16 of its 23 references that are in the map. First posted on arXiv 18 Jun 2024.

Builds on

16 of 23 references.

Closest ancestors · 6

Same direction, newest first

  1. 2024Testing the Limits of Jailbreaking Defenses with the Purple Problem20 Mar
  2. 2024JailbreakRadar: Comprehensive Assessment of Jailbreak Attacks Against LLMs8 Feb
  3. 2023Universal and Transferable Adversarial Attacks on Aligned Language Models27 Jul
  4. 2023MasterKey: Automated Jailbreak Across Multiple Large Language Model Chatbots16 Jul
  5. 2023Jailbroken: How Does LLM Safety Training Fail?5 Jul
  6. 2023Tricking LLMs into Disobedience: Formalizing, Analyzing, and Detecting Jailbreaks24 May

Other references · 10

  1. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  2. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  3. 2020Beyond English-Centric Multilingual Machine Translationcited by 559
  4. 2023Can AI-Generated Text be Reliably Detected?cited by 372
  5. 2023MEGA: Multilingual Evaluation of Generative AIcited by 291
  6. 2023Inverse Scaling: When Bigger Isn't Bettercited by 205
  7. 2020On the Computational Power of Transformers and its Implications in Sequence Modelingcited by 85
  8. 2023MEGAVERSE: Benchmarking Large Language Models Across Languages, Modalities, Models and Taskscited by 57
  9. 2023Tamil-Llama: A New Tamil Language Model Based on Llama 2cited by 31
  10. 2023Bergeron: Combating Adversarial Attacks through a Conscience-Based Alignment Frameworkcited by 29

Built on by

Cited by year

Citing papers per year: 2025: 2.

2 papers.

  1. 2025CourtGuard: A Local, Multiagent Prompt Injection Classifiercited by 1
  2. 2025A Real-Time, Self-Tuning Moderator Framework for Adversarial Prompt Detectioncited by 0