mlsanityData to 7 Oct 2026

Separate the Wheat from the Chaff: A Post-Hoc Approach to Safety Re-Alignment for Fine-Tuned Language Models

arXiv 2412.11041 ↗First posted 15 Dec 2024cs.CLcited by 19

On the map: LLM alignment / RLHF / preference optimisation → Safety alignment and jailbreak attacks

also in 1 more
Abstract (arXiv)

Although large language models (LLMs) achieve effective safety alignment at the time of release, they still face various safety challenges. A key issue is that fine-tuning often compromises the safety alignment of LLMs. To address this issue, we propose a method named IRR (Identify, Remove, and Recalibrate for Safety Realignment) that performs safety realignment for LLMs. The core of IRR is to identify and remove unsafe delta parameters from the fine-tuned models, while recalibrating the retained ones. We evaluate the effectiveness of IRR across various datasets, including both full fine-tuning and LoRA methods. Our results demonstrate that IRR significantly enhances the safety performance of fine-tuned models on safety benchmarks, such as harmful queries and jailbreak attacks, while maintaining their performance on downstream tasks. The source code is available at: https://anonymous.4open.science/r/IRR-BD4F.

Key facts. Cited by 19 papers in the map (data through 7 Oct 2026). Placed in Safety alignment and jailbreak attacks, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 41 of its 55 references that are in the map. First posted on arXiv 15 Dec 2024.

Builds on

41 of 55 references.

Closest ancestors · 10

Same direction, newest first

  1. 2024Harmful Fine-tuning Attacks and Defenses for Large Language Models: A Survey26 Sep
  2. 2024Antidote: Post-fine-tuning Safety Alignment for Large Language Models against Harmful Fine-tuning18 Aug
  3. 2024Safety Arithmetic: A Framework for Test-time Safety Alignment of Language Models by Steering Parameters and Activations17 Jun
  4. 2024Lisa: Lazy Safety Alignment for Large Language Models against Harmful Fine-tuning Attack28 May
  5. 2024Language Models are Homer Simpson! Safety Re-Alignment of Fine-tuned Language Models through Task Arithmetic19 Feb
  6. 2024Assessing the Brittleness of Safety Alignment via Pruning and Low-Rank Modifications7 Feb
  7. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!5 Oct
  8. 2023Shadow Alignment: The Ease of Subverting Safely-Aligned Language Models4 Oct
  9. 2023Universal and Transferable Adversarial Attacks on Aligned Language Models27 Jul
  10. 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Dataset10 Jul

Other references · 31

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  6. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  7. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  8. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  9. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,250
  10. 2024LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Modelscited by 1,442
  11. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,090
  12. 2023AGIEval: A Human-Centric Benchmark for Evaluating Foundation Modelscited by 701
19 more
  1. 2023Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunchcited by 679
  2. 2017Program Induction by Rationale Generation : Learning to Solve and Explain Algebraic Word Problemscited by 642
  3. 2021Merging Models with Fisher-Weighted Averagingcited by 613
  4. 2017To prune, or not to prune: exploring the efficacy of pruning for model compressioncited by 596
  5. 2023GPTFUZZER: Red Teaming Large Language Models with Auto-Generated Jailbreak Promptscited by 520
  6. 2023Is ChatGPT a General-Purpose Natural Language Processing Task Solver?cited by 486
  7. 2023MAmmoTH: Building Math Generalist Models through Hybrid Instruction Tuningcited by 483
  8. 2017A Survey of Model Compression and Acceleration for Deep Neural Networkscited by 298
  9. 2021Pruning and Quantization for Deep Neural Network Acceleration: A Surveycited by 259
  10. 2023Poisoning Language Models During Instruction Tuningcited by 254
  11. 2024SafeDecoding: Defending against Jailbreak Attacks via Safety-Aware Decodingcited by 233
  12. 2024SALAD-Bench: A Hierarchical and Comprehensive Safety Benchmark for Large Language Modelscited by 225
  13. 2021Group Fisher Pruning for Practical Network Compressioncited by 130
  14. 2024Safe LoRA: the Silver Lining of Reducing Safety Risks when Fine-tuning Large Language Modelscited by 122
  15. 2022NumGLUE: A Suite of Fundamental yet Challenging Mathematical Reasoning Taskscited by 110
  16. 2023Task-Specific Skill Localization in Fine-tuned Language Modelscited by 104
  17. 2024Model Tailor: Mitigating Catastrophic Forgetting in Multi-modal Large Language Modelscited by 54
  18. 2024Localize-and-Stitch: Efficient Model Merging via Sparse Task Arithmeticcited by 50
  19. 2023Language Model Unalignment: Parametric Red-Teaming to Expose Hidden Harms and Biasescited by 21

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 12; 2026: 6.

19 papers.

Sort:

By area:

  1. 2025A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deploymentcited by 110
  2. 2025Benign Samples Matter! Fine-tuning On Outlier Benign Samples Severely Breaks Safetycited by 22
  3. 2025Panacea: Mitigating Harmful Fine-tuning for Large Language Models via Post-fine-tuning Perturbationcited by 20
  4. 2025Are Smarter LLMs Safer? Exploring Safety-Reasoning Trade-offs in Prompting and Fine-Tuningcited by 19
  5. 2025Why LLM Safety Guardrails Collapse After Fine-tuning: A Similarity Analysis Between Alignment and Fine-tuning Datasetscited by 18
  6. 2025On Almost Surely Safe Alignment of Large Language Models at Inference-Timecited by 13
  7. 2025Adaptive Defense against Harmful Fine-Tuning for Large Language Models via Bayesian Data Schedulercited by 12
  8. 2024Safeguard Fine-Tuned LLMs Through Pre- and Post-Tuning Model Mergingcited by 11
  9. 2025CTRAP: Embedding Collapse Trap to Safeguard Large Language Models from Harmful Fine-Tuningcited by 11
  10. 2025Gradient Surgery for Safe LLM Fine-Tuningcited by 11
  11. 2025Safety Subspaces are Not Linearly Distinct: A Fine-Tuning Case Studycited by 6
  12. 2025Anchoring Refusal Direction: Mitigating Safety Risks in Tuning via Projection Constraintcited by 6
7 more
  1. 2025No, of Course I Can! Deeper Fine-Tuning Attacks That Bypass Token-Level Safety Mechanismscited by 5
  2. 2026Surgery: Mitigating Harmful Fine-Tuning for Large Language Models via Attention Sinkcited by 3
  3. 2026Safety Geometry Collapse in Multimodal LLMs and Adaptive Drift Correctioncited by 1
  4. 2026Two to Tango: Coupled Task-Reference Selection for Safe LLM Fine-tuningcited by 0
  5. 2026Deferred Exposure of Future Trajectories for Verifiable Reasoning in Autonomous Driving VLMscited by 0
  6. 2026CLEAR: Continuous Latent Adapter Routing for Utility-Preserving LLM Safety Alignmentcited by 0
  7. 2026SafeTune: A Unified Faithful Library for Auditing and Repairing Safety Drift in Fine-Tuned LLMscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Safety at One Shot: Patching Fine-Tuned LLMs with A Single Instance
  2. 2024Towards Comprehensive Post Safety Alignment of Large Language Models via Safety Patching
  3. 2025SafeMERGE: Preserving Safety Alignment in Fine-Tuned Large Language Models via Selective Layer-Wise Model Merging
  4. 2026Aligned but Fragile: Enhancing LLM Safety Robustness via Zeroth-Order Optimization
  5. 2025Fine-Tuning Lowers Safety and Disrupts Evaluation Consistency
  6. 2024Locking Down the Finetuned LLMs Safety
  7. 2024Safety Alignment Should Be Made More Than Just a Few Tokens Deep
  8. 2026LSSF: Safety Alignment for Large Language Models through Low-Rank Safety Subspace Fusion