mlsanityData to 7 Oct 2026

Backtracking Improves Generation Safety

arXiv 2409.14586 ↗First posted 22 Sep 2024cs.LG cs.AI cs.CLcited by 31

On the map: LLM alignment / RLHF / preference optimisation → Safety alignment and jailbreak attacks

also in 1 more
Abstract (arXiv)

Text generation has a fundamental limitation almost by definition: there is no taking back tokens that have been generated, even when they are clearly problematic. In the context of language model safety, when a partial unsafe generation is produced, language models by their nature tend to happily keep on generating similarly unsafe additional text. This is in fact how safety alignment of frontier models gets circumvented in the wild, despite great efforts in improving their safety. Deviating from the paradigm of approaching safety alignment as prevention (decreasing the probability of harmful responses), we propose backtracking, a technique that allows language models to "undo" and recover from their own unsafe generation through the introduction of a special [RESET] token. Our method can be incorporated into either SFT or DPO training to optimize helpfulness and harmlessness. We show that models trained to backtrack are consistently safer than baseline models: backtracking Llama-3-8B is four times more safe than the baseline model (6.1% → 1.5%) in our evaluations without regression in helpfulness. Our method additionally provides protection against four adversarial attacks including an adaptive attack, despite not being trained to do so.

Key facts. Cited by 31 papers in the map (data through 7 Oct 2026). Placed in Safety alignment and jailbreak attacks, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 48 of its 52 references that are in the map. First posted on arXiv 22 Sep 2024.

Builds on

48 of 52 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024Course-Correction: Safety Alignment Using Synthetic Preferences23 Jul
  2. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedback19 Oct
  3. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!5 Oct
  4. 2023AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Models3 Oct
  5. 2023Universal and Transferable Adversarial Attacks on Aligned Language Models27 Jul

Other references · 43

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  6. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
  7. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  8. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  9. 2017Deep reinforcement learning from human preferencescited by 3,846
  10. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  11. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  12. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,317
31 more
  1. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  2. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  3. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,722
  4. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,565
  5. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  6. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,330
  7. 2023Representation Engineering: A Top-Down Approach to AI Transparencycited by 1,239
  8. 2023Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversationscited by 1,124
  9. 2023Steering Llama 2 via Contrastive Activation Additioncited by 1,009
  10. 2022Red Teaming Language Models with Language Modelscited by 983
  11. 2023A General Theoretical Paradigm to Understand Learning from Human Preferencescited by 906
  12. 2023OpenAssistant Conversations -- Democratizing Large Language Model Alignmentcited by 712
  13. 2024Self-Rewarding Language Modelscited by 654
  14. 2024Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attackscited by 450
  15. 2019On Evaluating Adversarial Robustnesscited by 442
  16. 2023Catastrophic Jailbreak of Open-source LLMs via Exploiting Generationcited by 439
  17. 2023Safety-Tuned LLaMAs: Lessons From Improving the Safety of Large Language Models that Follow Instructionscited by 377
  18. 2022Self-critiquing models for assisting human evaluatorscited by 372
  19. 2024A StrongREJECT for Empty Jailbreakscited by 352
  20. 2024Improving Alignment and Robustness with Circuit Breakerscited by 302
  21. 2022Generating Sequences by Learning to Self-Correctcited by 271
  22. 2023Inspecting and Editing Knowledge Representations in Language Modelscited by 140
  23. 2024Tamper-Resistant Safeguards for Open-Weight LLMscited by 132
  24. 2023Do Language Models Know When They're Hallucinating References?cited by 115
  25. 2023LLMs cannot find reasoning errors, but can correct them given the error locationcited by 85
  26. 2022Language Generation Models Can Cause Harm: So What Can We Do About It? An Actionable Surveycited by 80
  27. 2023SimpleSafetyTests: a Test Suite for Identifying Critical Safety Risks in Large Language Modelscited by 72
  28. 2022Self-Destructing Models: Increasing the Costs of Harmful Dual Uses of Foundation Modelscited by 56
  29. 2023Bypassing the Safety Training of Open-Source LLMs with Priming Attackscited by 48
  30. 2021LM-Critic: Language Models for Unsupervised Grammatical Error Correctioncited by 33
  31. 2023SequenceMatch: Imitation Learning for Autoregressive Sequence Modelling with Backtrackingcited by 13

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 20; 2026: 7.

31 papers.

Sort:

By area:

  1. 2024Deliberative Alignment: Reasoning Enables Safer Language Modelscited by 292
  2. 2024Llama Guard 3 Vision: Safeguarding Human-AI Image Understanding Conversationscited by 118
  3. 2025Towards System 2 Reasoning in LLMs: Learning How to Think With Meta Chain-of-Thoughtcited by 108
  4. 2025FalseReject: A Resource for Improving Contextual Safety and Mitigating Over-Refusals in LLMs via Structured Reasoningcited by 53
  5. 2025AlphaSteer: Learning Refusal Steering with Principled Null-Space Constraintcited by 45
  6. 2024Controllable Safety Alignment: Inference-Time Adaptation to Diverse Safety Requirementscited by 35
  7. 2025ReVISE: Learning to Refine at Test-Time via Intrinsic Self-Verificationcited by 35
  8. 2025A Survey of Direct Preference Optimizationcited by 35
  9. 2025Step Back to Leap Forward: Self-Backtracking for Boosting Reasoning of Language Modelscited by 27
  10. 2025Reasoning as an Adaptive Defense for Safetycited by 22
  11. 2025SafeKey: Amplifying Aha-Moment Insights for Safety Reasoningcited by 20
  12. 2025Disentangling Reasoning and Knowledge in Medical Large Language Modelscited by 17
19 more
  1. 2025LongPerceptualThoughts: Distilling System-2 Reasoning for System-1 Perceptioncited by 14
  2. 2025AlphaAlign: Incentivizing Safety Alignment with Extremely Simplified Reinforcement Learningcited by 14
  3. 2025Large Reasoning Models Learn Better Alignment from Flawed Thinkingcited by 13
  4. 2025Towards Safe Reasoning in Large Reasoning Models via Corrective Interventioncited by 11
  5. 2025MetaDefense: Defending Finetuning-based Jailbreak Attack Before and During Generationcited by 8
  6. 2026LLMs Should Express Uncertainty Explicitlycited by 6
  7. 2025Reasoned Safety Alignment: Ensuring Jailbreak Defense via Answer-Then-Checkcited by 5
  8. 2024Jailbreaking LLM-Controlled Robotscited by 4
  9. 2025SAFER: Advancing Safety Alignment via Efficient Ex-Ante Reasoningcited by 4
  10. 2025Evaluating Language Model Reasoning about Confidential Informationcited by 3
  11. 2025Scalable and Robust LLM Unlearning by Correcting Responses with Retrieved Exclusionscited by 2
  12. 2025Antislop: A Comprehensive Framework for Identifying and Eliminating Repetitive Patterns in Language Modelscited by 1
  13. 2026Self-ReSET: Learning to Self-Recover from Unsafe Reasoning Trajectoriescited by 1
  14. 2026Conformal Certification of Reasoning Trace Prefixescited by 1
  15. 2026ALIGNBEAM : Inference-Time Alignment Transfer via Cross-Vocabulary Logit Mixingcited by 1
  16. 2025Matching Ranks Over Probability Yields Truly Deep Safety Alignmentcited by 0
  17. 2026Reinforcement Learning with Backtracking Feedbackcited by 0
  18. 2026Beyond Safe Data: Pretraining-Stage Alignment with Regular Safety Reflectioncited by 0
  19. 2026Addressing Over-Refusal in LLMs with Competing Rewardscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Backtracking for Safety
  2. 2024Simulate and Eliminate: Revoke Backdoors for Generative Large Language Models
  3. 2024BackdoorLLM: A Comprehensive Benchmark for Backdoor Attacks and Defenses on Large Language Models
  4. 2026Inference-Time Vulnerability Beyond Shallow Safety: Alignment Along Generation Trajectories
  5. 2024Safety Alignment Should Be Made More Than Just a Few Tokens Deep
  6. 2026Safety Paradox: How Enhanced Safety Awareness Leaves LLMs Vulnerable to Posterior Attack
  7. 2025DualEdit: Mitigating Safety Fallback in LLM Backdoor Editing via Affirmation-Refusal Regulation
  8. 2024No Two Devils Alike: Unveiling Distinct Mechanisms of Fine-tuning Attacks