mlsanityData to 14 Oct 2026

Mission Impossible: A Statistical Perspective on Jailbreaking LLMs

arXiv 2408.01420 ↗First posted 2 Aug 2024cs.LG cs.AI cs.CLcited by 24

On the map: LLM alignment / RLHF / preference optimisation → Safety alignment and jailbreak attacks

also in 1 more
Abstract (arXiv)

Large language models (LLMs) are trained on a deluge of text data with limited quality control. As a result, LLMs can exhibit unintended or even harmful behaviours, such as leaking information, fake news or hate speech. Countermeasures, commonly referred to as preference alignment, include fine-tuning the pretrained LLMs with carefully crafted text examples of desired behaviour. Even then, empirical evidence shows preference aligned LLMs can be enticed to harmful behaviour. This so called jailbreaking of LLMs is typically achieved by adversarially modifying the input prompt to the LLM. Our paper provides theoretical insights into the phenomenon of preference alignment and jailbreaking from a statistical perspective. Under our framework, we first show that pretrained LLMs will mimic harmful behaviour if present in the training corpus. Under that same framework, we then introduce a statistical notion of alignment, and lower-bound the jailbreaking probability, showing that it is unpreventable under reasonable assumptions. Based on our insights, we propose an alteration to the currently prevalent alignment strategy RLHF. Specifically, we introduce a simple modification to the RLHF objective, we call E-RLHF, that aims to increase the likelihood of safe responses. E-RLHF brings no additional training cost, and is compatible with other methods. Empirically, we demonstrate that E-RLHF outperforms RLHF on all alignment problems put forward by the AdvBench and HarmBench project without sacrificing model performance as measured by the MT-Bench project.

Key facts. Cited by 24 papers in the map (data through 14 Oct 2026). Placed in Safety alignment and jailbreak attacks, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 109 of its 130 references that are in the map. First posted on arXiv 2 Aug 2024.

Builds on

109 of 130 references.

Closest ancestors · 15

Same direction, newest first

  1. 2024DrAttack: Prompt Decomposition and Reconstruction Makes Powerful LLM Jailbreakers25 Feb
  2. 2024Defending Large Language Models against Jailbreak Attacks via Semantic Smoothing25 Feb
  3. 2024Assessing the Brittleness of Safety Alignment via Pruning and Low-Rank Modifications7 Feb
  4. 2024A Mechanistic Understanding of Alignment Algorithms: A Case Study on DPO and Toxicity3 Jan
  5. 2023Removing RLHF Protections in GPT-4 via Fine-Tuning9 Nov
  6. 2023Jailbreak and Guard Aligned Language Models with Only Few In-Context Demonstrations10 Oct
  7. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!5 Oct
  8. 2023Shadow Alignment: The Ease of Subverting Safely-Aligned Language Models4 Oct
  9. 2023AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Models3 Oct
  10. 2023Defending Against Alignment-Breaking Attacks via Robustly Aligned LLM18 Sep
  11. 2023Baseline Defenses for Adversarial Attacks Against Aligned Language Models1 Sep
  12. 2023GPT-4 Is Too Smart To Be Safe: Stealthy Chat with LLMs via Cipher12 Aug
3 more
  1. 2023Universal and Transferable Adversarial Attacks on Aligned Language Models27 Jul
  2. 2023Are aligned neural networks adversarially aligned?26 Jun
  3. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned23 Aug

Other references · 94

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2017Proximal Policy Optimization Algorithmscited by 15,215
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  5. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  6. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  7. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  8. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,415
  9. 2013Intriguing properties of neural networkscited by 5,155
  10. 2017Deep reinforcement learning from human preferencescited by 3,912
  11. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  12. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,211
38 more
  1. 2023Mistral 7Bcited by 3,164
  2. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,050
  3. 2016Deep Learning with Differential Privacycited by 2,728
  4. 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
  5. 2023Code Llama: Open Foundation Models for Codecited by 2,139
  6. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,998
  7. 2022Large Language Models Encode Clinical Knowledgecited by 1,830
  8. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,651
  9. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,429
  10. 2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,420
  11. 2019Certified Adversarial Robustness via Randomized Smoothingcited by 1,340
  12. 2023Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversationscited by 1,207
  13. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,055
  14. 2022Red Teaming Language Models with Language Modelscited by 1,026
  15. 2018Robustness May Be at Odds with Accuracycited by 880
  16. 2023Large Language Models as Optimizerscited by 864
  17. 2022Ignore Previous Prompt: Attack Techniques For Language Modelscited by 832
  18. 2023A Simple and Effective Pruning Approach for Large Language Modelscited by 800
  19. 2019Universal Adversarial Triggers for Attacking and Analyzing NLPcited by 722
  20. 2023Tree of Attacks: Jailbreaking Black-Box LLMs Automaticallycited by 672
  21. 2017Exploring Generalization in Deep Learningcited by 600
  22. 2024How Johnny Can Persuade LLMs to Jailbreak Them: Rethinking Persuasion to Challenge AI Safety by Humanizing LLMscited by 573
  23. 2019A Simple Baseline for Bayesian Uncertainty in Deep Learningcited by 560
  24. 2023GPTFUZZER: Red Teaming Large Language Models with Auto-Generated Jailbreak Promptscited by 560
  25. 2023Jailbreaking ChatGPT via Prompt Engineering: An Empirical Studycited by 498
  26. 2023"Do Anything Now": Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Modelscited by 489
  27. 2023SmoothLLM: Defending Large Language Models Against Jailbreaking Attackscited by 469
  28. 2023Catastrophic Jailbreak of Open-source LLMs via Exploiting Generationcited by 451
  29. 2023Detecting Language Model Attacks with Perplexitycited by 431
  30. 2023LeanDojo: Theorem Proving with Retrieval-Augmented Language Modelscited by 427
  31. 2023DeepInception: Hypnotize Large Language Model to Be Jailbreakercited by 367
  32. 2023Hard Prompts Made Easy: Gradient-Based Discrete Optimization for Prompt Tuning and Discoverycited by 336
  33. 2023Low-Resource Languages Jailbreak GPT-4cited by 309
  34. 2023Exploiting Programmatic Behavior of LLMs: Dual-Use Through Standard Security Attackscited by 278
  35. 2023Jailbreak in pieces: Compositional Adversarial Attacks on Multi-Modal Language Modelscited by 273
  36. 2023Certifying LLM Safety against Adversarial Promptingcited by 266
  37. 2023A Wolf in Sheep's Clothing: Generalized Nested Jailbreak Prompts can Fool Large Language Models Easilycited by 259
  38. 2023Pretraining Language Models with Human Preferencescited by 247

50 most cited of 94 All 94 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 5; 2025: 12; 2026: 7.

24 papers.

Sort:

By area:

  1. 2025The Hidden Dimensions of LLM Alignment: A Multi-Dimensional Analysis of Orthogonal Safety Directionscited by 41
  2. 2024Securing Large Language Models: Addressing Bias, Misinformation, and Prompt Attackscited by 26
  3. 2025On Almost Surely Safe Alignment of Large Language Models at Inference-Timecited by 14
  4. 2025The Geometry of Harmfulness in LLMs through Subconcept Probingcited by 7
  5. 2024Recent Advances in Attack and Defense Approaches of Large Language Modelscited by 6
  6. 2024Sample Compression Unleashed: New Generalization Bounds for Real Valued Lossescited by 4
  7. 2024LIAR: Leveraging Inference Time Alignment (Best-of-N) to Jailbreak LLMs in Secondscited by 4
  8. 2025Efficient Safety Retrofitting Against Jailbreaking for LLMscited by 4
  9. 2026Jailbreak Scaling Laws for Large Language Models: Polynomial-Exponential Crossovercited by 4
  10. 2025Calibrated Predictive Lower Bounds on Time-to-Unsafe-Sampling in LLMscited by 3
  11. 2025Do Internal Layers of LLMs Reveal Patterns for Jailbreak Detection?cited by 3
  12. 2024Gen-AI for User Safety: A Surveycited by 2
12 more
  1. 2026Bound to Disagree: Generalization Bounds via Certifiable Surrogatescited by 2
  2. 2026Analyzing Defensive Misdirection Against Model-Guided Automated Attacks on Agentic AI Systemscited by 2
  3. 2025LLM in the Middle: A Systematic Review of Threats and Mitigations to Real-World LLM-based Systemscited by 1
  4. 2025GEO-Detective: Unveiling Location Privacy Risks in Images with LLM Agentscited by 1
  5. 2026A Theoretical Game of Attacks via Compositional Skillscited by 1
  6. 2026SoK: Intent-Oriented Systematization of Multi-Turn LLM Jailbreakscited by 1
  7. 2025Concealment of Intent: A Game-Theoretic Analysiscited by 0
  8. 2025LLM Reinforcement in Contextcited by 0
  9. 2025TASO: Jailbreak LLMs via Alternative Template and Suffix Optimizationcited by 0
  10. 2025Invasive Context Engineering to Control Large Language Modelscited by 0
  11. 2026Jailbreaking LLMs via Calibrationcited by 0
  12. 2026Monotonicity as an Architectural Bias for Robust Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Jailbreaking Large Language Models Through Alignment Vulnerabilities in Out-of-Distribution Settings
  2. 2025Playing the Fool: Jailbreaking LLMs and Multimodal LLMs with Out-of-Distribution Strategy
  3. 2025A Domain-Based Taxonomy of Jailbreak Vulnerabilities in Large Language Models
  4. 2025Uncovering the Persuasive Fingerprint of LLMs in Jailbreaking Attacks
  5. 2024Characterizing and Evaluating the Reliability of LLMs against Jailbreak Attacks
  6. 2024Defending against Jailbreak through Early Exit Generation of Large Language Models
  7. 2025Scaling Patterns in Adversarial Alignment: Evidence from Multi-LLM Jailbreak Experiments
  8. 2026Why Do Aligned LLMs Remain Jailbreakable: Refusal-Escape Directions, Operator-Level Sources, and Safety-Utility Trade-off