mlsanityData to 14 Oct 2026

Safety at One Shot: Patching Fine-Tuned LLMs with A Single Instance

arXiv 2601.01887 ↗First posted 5 Jan 2026cs.LG cs.AIcited by 8

On the map: LLM alignment / RLHF / preference optimisation → Safety alignment and jailbreak attacks

Abstract (arXiv)

Fine-tuning safety-aligned large language models (LLMs) can substantially compromise their safety. Previous approaches require many safety samples or calibration sets, which not only incur significant computational overhead during realignment but also lead to noticeable degradation in model utility. Contrary to this belief, we show that safety alignment can be fully recovered with only a single safety example, without sacrificing utility and at minimal cost. Remarkably, this recovery is effective regardless of the number of harmful examples used in fine-tuning or the size of the underlying model, and convergence is achieved within just a few epochs. Furthermore, we uncover the low-rank structure of the safety gradient, which explains why such efficient correction is possible. We validate our findings across five safety-aligned LLMs and multiple datasets, demonstrating the generality of our approach.

Key facts. Cited by 8 papers in the map (data through 14 Oct 2026). Placed in Safety alignment and jailbreak attacks, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 57 of its 66 references that are in the map. First posted on arXiv 5 Jan 2026.

Builds on

57 of 66 references.

Closest ancestors · 16

Same direction, newest first

  1. 2025Shape it Up! Restoring LLM Safety during Finetuning22 May
  2. 2024Antidote: Post-fine-tuning Safety Alignment for Large Language Models against Harmful Fine-tuning18 Aug
  3. 2024PKU-SafeRLHF: Towards Multi-Level Safety Alignment for LLMs with Human Preference20 Jun
  4. 2024Safety Alignment Should Be Made More Than Just a Few Tokens Deep10 Jun
  5. 2024Lisa: Lazy Safety Alignment for Large Language Models against Harmful Fine-tuning Attack28 May
  6. 2024Navigating the Safety Landscape: Measuring Risks in Finetuning Large Language Models27 May
  7. 2024No Two Devils Alike: Unveiling Distinct Mechanisms of Fine-tuning Attacks25 May
  8. 2024Assessing the Brittleness of Safety Alignment via Pruning and Low-Rank Modifications7 Feb
  9. 2024Vaccine: Perturbation-aware Alignment for Large Language Models against Harmful Fine-tuning Attack2 Feb
  10. 2023LoRA Fine-tuning Efficiently Undoes Safety Training in Llama 2-Chat 70B31 Oct
  11. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedback19 Oct
  12. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!5 Oct
4 more
  1. 2023Shadow Alignment: The Ease of Subverting Safely-Aligned Language Models4 Oct
  2. 2023Universal and Transferable Adversarial Attacks on Aligned Language Models27 Jul
  3. 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Dataset10 Jul
  4. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned23 Aug

Other references · 41

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2024The Llama 3 Herd of Modelscited by 14,715
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  5. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  6. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  7. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  8. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  9. 2024Qwen2.5 Technical Reportcited by 4,942
  10. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  11. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  12. 2023Mistral 7Bcited by 3,164
29 more
  1. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,429
  2. 2020Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuningcited by 671
  3. 2016Linear Convergence of Gradient and Proximal-Gradient Methods Under the Polyak-Lojasiewicz Conditioncited by 539
  4. 2019SAMSum Corpus: A Human-annotated Dialogue Dataset for Abstractive Summarizationcited by 535
  5. 2024LESS: Selecting Influential Data for Targeted Instruction Tuningcited by 532
  6. 2023DecodingTrust: A Comprehensive Assessment of Trustworthiness in GPT Modelscited by 518
  7. 2023Jailbreaking ChatGPT via Prompt Engineering: An Empirical Studycited by 498
  8. 2018Bilevel Programming for Hyperparameter Optimization and Meta-Learningcited by 405
  9. 2023Fine-Tuning Language Models with Just Forward Passescited by 345
  10. 2023Data Selection for Language Models via Importance Resamplingcited by 329
  11. 2024Attacks, Defenses and Evaluations for LLM Conversation Safety: A Surveycited by 130
  12. 2023#InsTag: Instruction Tagging for Analyzing Supervised Fine-tuning of Large Language Modelscited by 109
  13. 2022BOME! Bilevel Optimization Made Easy: A Simple First-Order Approachcited by 105
  14. 2024DsDm: Model-Aware Dataset Selection with Datamodelscited by 105
  15. 2023On Penalty-based Bilevel Gradient Descent Methodcited by 87
  16. 2024Long Is More for Alignment: A Simple but Tough-to-Beat Baseline for Instruction Fine-Tuningcited by 85
  17. 2020A Generic First-Order Algorithmic Framework for Bi-Level Programming Beyond Lower-Level Singletoncited by 79
  18. 2017A First Order Method for Solving Convex Bi-Level Optimization Problemscited by 67
  19. 2024SEAL: Safety-enhanced Aligned LLM Fine-tuning via Bilevel Data Selectioncited by 64
  20. 2024What Makes and Breaks Safety Fine-tuning? A Mechanistic Studycited by 55
  21. 2023On Penalty Methods for Nonconvex Bilevel Optimization and First-Order Stochastic Approximationcited by 51
  22. 2023First-order penalty methods for bilevel optimizationcited by 41
  23. 2024Principled Penalty-based Methods for Bilevel Reinforcement Learning and RLHFcited by 41
  24. 2025Model Tampering Attacks Enable More Rigorous Evaluations of LLM Capabilitiescited by 39
  25. 2024ScaleBiO: Scalable Bilevel Optimization for LLM Data Reweightingcited by 37
  26. 2024Get more for less: Principled Data Selection for Warming Up Fine-Tuning in LLMscited by 32
  27. 2022Alternating Implicit Projected SGD and Its Efficient Variants for Equality-constrained Bilevel Optimizationcited by 10
  28. 2025Alleviating the Fear of Losing Alignment in LLM Fine-tuningcited by 9
  29. 2025Activation Approximations Can Incur Safety Vulnerabilities Even in Aligned LLMs: Comprehensive Analysis and Defensecited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 8.

8 papers.

  1. 2026Surgery: Mitigating Harmful Fine-Tuning for Large Language Models via Attention Sinkcited by 4
  2. 2026Alignment Dynamics in LLM Fine-Tuningcited by 2
  3. 2026Patcher: Post-Hoc Patching of Backdoored Large Language Modelscited by 2
  4. 2026Mitigating Many-shot Jailbreak Attacks with One Single Demonstrationcited by 0
  5. 2026Defending Against Harmful Supervision Hidden in Benign Samplescited by 0
  6. 2026HyperSafe: Inference-Time Safety Recovery for Fine-Tuned Language Modelscited by 0
  7. 2026TRACE: Trajectory-Based Safety Patch Learning for LLM Post-Training Realignmentcited by 0
  8. 2026SafeTune: A Unified Faithful Library for Auditing and Repairing Safety Drift in Fine-Tuned LLMscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Towards Comprehensive Post Safety Alignment of Large Language Models via Safety Patching
  2. 2026Aligned but Fragile: Enhancing LLM Safety Robustness via Zeroth-Order Optimization
  3. 2026LSSF: Safety Alignment for Large Language Models through Low-Rank Safety Subspace Fusion
  4. 2024Separate the Wheat from the Chaff: A Post-Hoc Approach to Safety Re-Alignment for Fine-Tuned Language Models
  5. 2026Safeguarding LLM Fine-tuning via Push-Pull Distributional Alignment
  6. 2024Locking Down the Finetuned LLMs Safety
  7. 2025SafeMERGE: Preserving Safety Alignment in Fine-Tuned Large Language Models via Selective Layer-Wise Model Merging
  8. 2024Safety Arithmetic: A Framework for Test-time Safety Alignment of Language Models by Steering Parameters and Activations