# Backtracking Improves Generation Safety

arXiv: https://arxiv.org/abs/2409.14586 · HTML: https://mlsanity.com/paper/2409.14586/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- Placed in Safety alignment and jailbreak attacks, one of 10 directions in LLM alignment / RLHF / preference optimisation.
- Builds on 48 of its 52 references that are in the map.
- First posted on arXiv 22 Sep 2024.

## Placed in

- LLM alignment / RLHF / preference optimisation / [Safety alignment and jailbreak attacks](https://mlsanity.com/alignment/safety-alignment-and-jailbreak-attacks/)
- LLM safety and security / [Safety alignment, refusal and guardrails](https://mlsanity.com/safety/safety-alignment-refusal-and-guardrails/)

## Builds on (most cited first, 30 of 48)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Towards Deep Learning Models Resistant to Adversarial Attacks](https://mlsanity.com/paper/1706.06083/) (2017)
- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://mlsanity.com/paper/2203.11171/) (2022)
- [Deep reinforcement learning from human preferences](https://mlsanity.com/paper/1706.03741/) (2017)
- [Finetuned Language Models Are Zero-Shot Learners](https://mlsanity.com/paper/2109.01652/) (2021)
- [Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2204.05862/) (2022)
- [Self-Refine: Iterative Refinement with Self-Feedback](https://mlsanity.com/paper/2303.17651/) (2023)
- [Universal and Transferable Adversarial Attacks on Aligned Language Models](https://mlsanity.com/paper/2307.15043/) (2023)
- [Constitutional AI: Harmlessness from AI Feedback](https://mlsanity.com/paper/2212.08073/) (2022)
- [The Curious Case of Neural Text Degeneration](https://mlsanity.com/paper/1904.09751/) (2019)
- [WebGPT: Browser-assisted question-answering with human feedback](https://mlsanity.com/paper/2112.09332/) (2021)
- [Jailbroken: How Does LLM Safety Training Fail?](https://mlsanity.com/paper/2307.02483/) (2023)
- [Multitask Prompted Training Enables Zero-Shot Task Generalization](https://mlsanity.com/paper/2110.08207/) (2021)
- [HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal](https://mlsanity.com/paper/2402.04249/) (2024)
- [Representation Engineering: A Top-Down Approach to AI Transparency](https://mlsanity.com/paper/2310.01405/) (2023)
- [Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversations](https://mlsanity.com/paper/2312.06674/) (2023)
- [Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!](https://mlsanity.com/paper/2310.03693/) (2023)
- [Steering Llama 2 via Contrastive Activation Addition](https://mlsanity.com/paper/2312.06681/) (2023)
- [Red Teaming Language Models with Language Models](https://mlsanity.com/paper/2202.03286/) (2022)
- [A General Theoretical Paradigm to Understand Learning from Human Preferences](https://mlsanity.com/paper/2310.12036/) (2023)
- [AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Models](https://mlsanity.com/paper/2310.04451/) (2023)
- [OpenAssistant Conversations -- Democratizing Large Language Model Alignment](https://mlsanity.com/paper/2304.07327/) (2023)
- [Self-Rewarding Language Models](https://mlsanity.com/paper/2401.10020/) (2024)
- [Safe RLHF: Safe Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2310.12773/) (2023)
- [Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attacks](https://mlsanity.com/paper/2404.02151/) (2024)

## Built on by (most cited first, 30 of 32)

- [Deliberative Alignment: Reasoning Enables Safer Language Models](https://mlsanity.com/paper/2412.16339/) (2024)
- [Llama Guard 3 Vision: Safeguarding Human-AI Image Understanding Conversations](https://mlsanity.com/paper/2411.10414/) (2024)
- [Towards System 2 Reasoning in LLMs: Learning How to Think With Meta Chain-of-Thought](https://mlsanity.com/paper/2501.04682/) (2025)
- [FalseReject: A Resource for Improving Contextual Safety and Mitigating Over-Refusals in LLMs via Structured Reasoning](https://mlsanity.com/paper/2505.08054/) (2025)
- [AlphaSteer: Learning Refusal Steering with Principled Null-Space Constraint](https://mlsanity.com/paper/2506.07022/) (2025)
- [Controllable Safety Alignment: Inference-Time Adaptation to Diverse Safety Requirements](https://mlsanity.com/paper/2410.08968/) (2024)
- [A Survey of Direct Preference Optimization](https://mlsanity.com/paper/2503.11701/) (2025)
- [ReVISE: Learning to Refine at Test-Time via Intrinsic Self-Verification](https://mlsanity.com/paper/2502.14565/) (2025)
- [Step Back to Leap Forward: Self-Backtracking for Boosting Reasoning of Language Models](https://mlsanity.com/paper/2502.04404/) (2025)
- [Reasoning as an Adaptive Defense for Safety](https://mlsanity.com/paper/2507.00971/) (2025)
- [SafeKey: Amplifying Aha-Moment Insights for Safety Reasoning](https://mlsanity.com/paper/2505.16186/) (2025)
- [Disentangling Reasoning and Knowledge in Medical Large Language Models](https://mlsanity.com/paper/2505.11462/) (2025)
- [LongPerceptualThoughts: Distilling System-2 Reasoning for System-1 Perception](https://mlsanity.com/paper/2504.15362/) (2025)
- [AlphaAlign: Incentivizing Safety Alignment with Extremely Simplified Reinforcement Learning](https://mlsanity.com/paper/2507.14987/) (2025)
- [Large Reasoning Models Learn Better Alignment from Flawed Thinking](https://mlsanity.com/paper/2510.00938/) (2025)
- [Towards Safe Reasoning in Large Reasoning Models via Corrective Intervention](https://mlsanity.com/paper/2509.24393/) (2025)
- [MetaDefense: Defending Finetuning-based Jailbreak Attack Before and During Generation](https://mlsanity.com/paper/2510.07835/) (2025)
- [Jailbreaking LLM-Controlled Robots](https://mlsanity.com/paper/2410.13691/) (2024)
- [Reasoned Safety Alignment: Ensuring Jailbreak Defense via Answer-Then-Check](https://mlsanity.com/paper/2509.11629/) (2025)
- [LLMs Should Express Uncertainty Explicitly](https://mlsanity.com/paper/2604.05306/) (2026)
- [StepGuard: Learning Step-Level Guardrails with Scalable Supervision and Safety-Utility Balancing](https://mlsanity.com/paper/2608.24777/) (2026)
- [SAFER: Advancing Safety Alignment via Efficient Ex-Ante Reasoning](https://mlsanity.com/paper/2504.02725/) (2025)
- [Evaluating Language Model Reasoning about Confidential Information](https://mlsanity.com/paper/2508.19980/) (2025)
- [Scalable and Robust LLM Unlearning by Correcting Responses with Retrieved Exclusions](https://mlsanity.com/paper/2509.25973/) (2025)
- [Conformal Certification of Reasoning Trace Prefixes](https://mlsanity.com/paper/2605.30085/) (2026)
- [Antislop: A Comprehensive Framework for Identifying and Eliminating Repetitive Patterns in Language Models](https://mlsanity.com/paper/2510.15061/) (2025)
- [Self-ReSET: Learning to Self-Recover from Unsafe Reasoning Trajectories](https://mlsanity.com/paper/2605.08936/) (2026)
- [ALIGNBEAM : Inference-Time Alignment Transfer via Cross-Vocabulary Logit Mixing](https://mlsanity.com/paper/2606.12342/) (2026)
- [Beyond Safe Data: Pretraining-Stage Alignment with Regular Safety Reflection](https://mlsanity.com/paper/2606.19168/) (2026)
- [Matching Ranks Over Probability Yields Truly Deep Safety Alignment](https://mlsanity.com/paper/2512.05518/) (2025)

## Abstract (arXiv)

Text generation has a fundamental limitation almost by definition: there is no taking back tokens that have been generated, even when they are clearly problematic. In the context of language model safety, when a partial unsafe generation is produced, language models by their nature tend to happily keep on generating similarly unsafe additional text. This is in fact how safety alignment of frontier models gets circumvented in the wild, despite great efforts in improving their safety. Deviating from the paradigm of approaching safety alignment as prevention (decreasing the probability of harmful responses), we propose backtracking, a technique that allows language models to "undo" and recover from their own unsafe generation through the introduction of a special [RESET] token. Our method can be incorporated into either SFT or DPO training to optimize helpfulness and harmlessness. We show that models trained to backtrack are consistently safer than baseline models: backtracking Llama-3-8B is four times more safe than the baseline model (6.1% → 1.5%) in our evaluations without regression in helpfulness. Our method additionally provides protection against four adversarial attacks including an adaptive attack, despite not being trained to do so.