# Break the Breakout: Reinventing LM Defense Against Jailbreak Attacks with Self-Refinement

arXiv: https://arxiv.org/abs/2402.15180 · HTML: https://mlsanity.com/paper/2402.15180/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- Placed in Jailbreaking, one of 12 directions in LLM safety and security.
- Builds on 19 of its 26 references that are in the map.
- First posted on arXiv 23 Feb 2024.

## Placed in

- LLM safety and security / [Jailbreaking](https://mlsanity.com/safety/jailbreaking/)

## Builds on (most cited first, 19 of 19)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Finetuned Language Models Are Zero-Shot Learners](https://mlsanity.com/paper/2109.01652/) (2021)
- [Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2204.05862/) (2022)
- [Self-Refine: Iterative Refinement with Self-Feedback](https://mlsanity.com/paper/2303.17651/) (2023)
- [Universal and Transferable Adversarial Attacks on Aligned Language Models](https://mlsanity.com/paper/2307.15043/) (2023)
- [Jailbroken: How Does LLM Safety Training Fail?](https://mlsanity.com/paper/2307.02483/) (2023)
- [BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Dataset](https://mlsanity.com/paper/2307.04657/) (2023)
- [Safe RLHF: Safe Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2310.12773/) (2023)
- [Zephyr: Direct Distillation of LM Alignment](https://mlsanity.com/paper/2310.16944/) (2023)
- [Jailbreaking ChatGPT via Prompt Engineering: An Empirical Study](https://mlsanity.com/paper/2305.13860/) (2023)
- [SmoothLLM: Defending Large Language Models Against Jailbreaking Attacks](https://mlsanity.com/paper/2310.03684/) (2023)
- [Knowledge Unlearning for Mitigating Privacy Risks in Language Models](https://mlsanity.com/paper/2210.01504/) (2022)
- [Jailbreak and Guard Aligned Language Models with Only Few In-Context Demonstrations](https://mlsanity.com/paper/2310.06387/) (2023)
- [OpenChat: Advancing Open-source Language Models with Mixed-Quality Data](https://mlsanity.com/paper/2309.11235/) (2023)
- [Red-Teaming Large Language Models using Chain of Utterances for Safety-Alignment](https://mlsanity.com/paper/2308.09662/) (2023)

## Built on by (most cited first, 29 of 29)

- [Jailbreak Attacks and Defenses Against Large Language Models: A Survey](https://mlsanity.com/paper/2407.04295/) (2024)
- [Know Your Limits: A Survey of Abstention in Large Language Models](https://mlsanity.com/paper/2407.18418/) (2024)
- [Defending Large Language Models against Jailbreak Attacks via Semantic Smoothing](https://mlsanity.com/paper/2402.16192/) (2024)
- [JailbreakZoo: Survey, Landscapes, and Horizons in Jailbreaking Large Language and Vision-Language Models](https://mlsanity.com/paper/2407.01599/) (2024)
- [BlueSuffix: Reinforced Blue Teaming for Vision-Language Models Against Jailbreak Attacks](https://mlsanity.com/paper/2410.20971/) (2024)
- [Monitoring Decomposition Attacks in LLMs with Lightweight Sequential Monitors](https://mlsanity.com/paper/2506.10949/) (2025)
- [From LLMs to MLLMs: Exploring the Landscape of Multimodal Jailbreaking](https://mlsanity.com/paper/2406.14859/) (2024)
- [Mission Impossible: A Statistical Perspective on Jailbreaking LLMs](https://mlsanity.com/paper/2408.01420/) (2024)
- [PAPILLON: Efficient and Stealthy Fuzz Testing-Powered Jailbreaks for LLMs](https://mlsanity.com/paper/2409.14866/) (2024)
- [Revisiting Multi-Agent Debate as Test-Time Scaling: A Systematic Study of Conditional Effectiveness](https://mlsanity.com/paper/2505.22960/) (2025)
- [Intrinsic Self-correction for Enhanced Morality: An Analysis of Internal Mechanisms and the Superficial Hypothesis](https://mlsanity.com/paper/2407.15286/) (2024)
- [AutoJailbreak: Exploring Jailbreak Attacks and Defenses through a Dependency Lens](https://mlsanity.com/paper/2406.03805/) (2024)
- [You Can't Eat Your Cake and Have It Too: The Performance Degradation of LLMs with Jailbreak Defense](https://mlsanity.com/paper/2501.12210/) (2025)
- [Single-pass Detection of Jailbreaking Input in Large Language Models](https://mlsanity.com/paper/2502.15435/) (2025)
- [Proactive defense against LLM Jailbreak](https://mlsanity.com/paper/2510.05052/) (2025)
- [PrisonBreak: Jailbreaking Large Language Models with at Most Twenty-Five Targeted Bit-flips](https://mlsanity.com/paper/2412.07192/) (2024)
- [SoK: Systematizing LLM Prompt Security: Taxonomies, Datasets, and Unified Evaluation of Attacks and Defenses](https://mlsanity.com/paper/2510.15476/) (2025)
- [Secure Tug-of-War (SecTOW): Iterative Defense-Attack Training with Reinforcement Learning for Multimodal Model Security](https://mlsanity.com/paper/2507.22037/) (2025)
- [Preventing Jailbreak Prompts as Malicious Tools for Cybercriminals: A Cyber Defense Perspective](https://mlsanity.com/paper/2411.16642/) (2024)
- [Can Small Language Models Reliably Resist Jailbreak Attacks? A Comprehensive Evaluation](https://mlsanity.com/paper/2503.06519/) (2025)
- [Jailbreaking LLMs & VLMs: Mechanisms, Evaluation, and Unified Defense](https://mlsanity.com/paper/2601.03594/) (2026)
- [Fail-Closed Alignment for Large Language Models](https://mlsanity.com/paper/2602.16977/) (2026)
- [Enhancing the Safety of Medical Vision-Language Models by Synthetic Demonstrations](https://mlsanity.com/paper/2506.09067/) (2025)
- [LoopTrap: Termination Poisoning Attacks on LLM Agents](https://mlsanity.com/paper/2605.05846/) (2026)
- [Dynamic Deep Prompt Optimization for Defending Against Jailbreak Attacks on LLMs](https://mlsanity.com/paper/2609.26185/) (2026)
- [SoK: a Comprehensive Causality Analysis Framework for Large Language Model Security](https://mlsanity.com/paper/2512.04841/) (2025)
- [A Systematic Study of Training-Free Methods for Trustworthy Large Language Models](https://mlsanity.com/paper/2604.15789/) (2026)
- [From Interpretability to Control: Insights from Six Years of the TrustNLP Workshop](https://mlsanity.com/paper/2608.11171/) (2026)
- [Reactivating Alignment: Defending LLMs from Jailbreaks via Intention-Aware Input-Output Matching](https://mlsanity.com/paper/2610.04470/) (2026)

## Abstract (arXiv)

Caution: This paper includes offensive words that could potentially cause unpleasantness. Language models (LMs) are vulnerable to exploitation for adversarial misuse. Training LMs for safety alignment is extensive and makes it hard to respond to fast-developing attacks immediately, such as jailbreaks. We propose self-refine with formatting that achieves outstanding safety even in non-safety-aligned LMs and evaluate our method alongside several defense baselines, demonstrating that it is the safest training-free method against jailbreak attacks. Additionally, we proposed a formatting method that improves the efficiency of the self-refine process while reducing attack success rates in fewer iterations. We've also observed that non-safety-aligned LMs outperform safety-aligned LMs in safety tasks by giving more helpful and safe responses. In conclusion, our findings can achieve less safety risk with fewer computational costs, allowing non-safety LM to be easily utilized in real-world service.