mlsanityData to 14 Oct 2026

Language Models are Homer Simpson! Safety Re-Alignment of Fine-tuned Language Models through Task Arithmetic

arXiv 2402.11746 ↗First posted 19 Feb 2024cs.CL cs.AIcited by 112

On the map: LLM alignment / RLHF / preference optimisation → Safety alignment and jailbreak attacks

Abstract (arXiv)

Aligned language models face a significant limitation as their fine-tuning often results in compromised safety. To tackle this, we propose a simple method RESTA that performs LLM safety realignment. RESTA stands for REstoring Safety through Task Arithmetic. At its core, it involves a simple arithmetic addition of a safety vector to the weights of the compromised model. We demonstrate the effectiveness of RESTA in both parameter-efficient and full fine-tuning, covering a wide range of downstream tasks, including instruction following in Chinese, English, and Hindi, as well as problem-solving capabilities in Code and Math. We also showcase the generalizability of RESTA on three existing safety evaluation benchmarks and a multilingual benchmark dataset proposed as a part of this work, consisting of 550 harmful questions covering 11 categories, each with 5 sub-categories of harm. Overall, RESTA decreases the harmfulness of the compromised model from 18.6% to 5.1% and from 9.2% to 1.5% in parameter-efficient and full fine-tuning, respectively, while maintaining most of the model's performance on the task. We release the source codes at: https://github.com/declare-lab/resta.

Key facts. Cited by 112 papers in the map (data through 14 Oct 2026). Placed in Safety alignment and jailbreak attacks, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 39 of its 49 references that are in the map. First posted on arXiv 19 Feb 2024.

Builds on

39 of 49 references.

Closest ancestors · 4

Same direction, newest first

  1. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!5 Oct
  2. 2023Shadow Alignment: The Ease of Subverting Safely-Aligned Language Models4 Oct
  3. 2023Universal and Transferable Adversarial Attacks on Aligned Language Models27 Jul
  4. 2023Are aligned neural networks adversarially aligned?26 Jun

Other references · 35

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  5. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  6. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  7. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  8. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  9. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  10. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,724
  11. 2023Code Llama: Open Foundation Models for Codecited by 2,139
  12. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,396
23 more
  1. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,245
  2. 2018Averaging Weights Leads to Wider Optima and Better Generalizationcited by 1,209
  3. 2018XNLI: Evaluating Cross-lingual Sentence Representationscited by 915
  4. 2021Robust fine-tuning of zero-shot modelscited by 797
  5. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 781
  6. 2023Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunchcited by 694
  7. 2019Linear Mode Connectivity and the Lottery Ticket Hypothesiscited by 637
  8. 2021Merging Models with Fisher-Weighted Averagingcited by 621
  9. 2020What is being transferred in transfer learning?cited by 368
  10. 2020XCOPA: A Multilingual Dataset for Causal Commonsense Reasoningcited by 333
  11. 2023Efficient and Effective Text Encoding for Chinese LLaMA and Alpacacited by 305
  12. 2023Red-Teaming Large Language Models using Chain of Utterances for Safety-Alignmentcited by 223
  13. 2022On Second Thought, Let's Not Think Step by Step! Bias and Toxicity in Zero-Shot Reasoningcited by 194
  14. 2022Branch-Train-Merge: Embarrassingly Parallel Training of Expert Language Modelscited by 191
  15. 2022Patching open-vocabulary models by interpolating weightscited by 190
  16. 2020Deep learning versus kernel learning: an empirical study of loss landscape geometry and the time evolution of the Neural Tangent Kernelcited by 187
  17. 2023RAIN: Your Language Models Can Align Themselves without Finetuningcited by 157
  18. 2023On the Exploitability of Instruction Tuningcited by 116
  19. 2022Fusing finetuned models for better pretrainingcited by 110
  20. 2022Self-Destructing Models: Increasing the Costs of Harmful Dual Uses of Foundation Modelscited by 57
  21. 2023OpinionGPT: Modelling Explicit Biases in Instruction-Tuned LLMscited by 29
  22. 2023Language Model Unalignment: Parametric Red-Teaming to Expose Hidden Harms and Biasescited by 24
  23. 2021KNOT: Knowledge Distillation using Optimal Transport for Solving NLP Taskscited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 28; 2025: 38; 2026: 46.

112 papers.

Sort:

By area:

  1. 2024LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Modelscited by 1,464
  2. 2024Model Merging in LLMs, MLLMs, and Beyond: Methods, Theories, Applications and Opportunitiescited by 254
  3. 2025Large Language Models Often Know When They Are Being Evaluatedcited by 128
  4. 2024Know Your Limits: A Survey of Abstention in Large Language Modelscited by 121
  5. 2024Booster: Tackling Harmful Fine-tuning for Large Language Models via Attenuating Harmful Perturbationcited by 92
  6. 2024Harmful Fine-tuning Attacks and Defenses for Large Language Models: A Surveycited by 90
  7. 2025LLMs Encode Harmfulness and Refusal Separatelycited by 88
  8. 2024SafetyPrompts: a Systematic Review of Open Datasets for Evaluating and Improving Large Language Model Safetycited by 77
  9. 2024Antidote: Post-fine-tuning Safety Alignment for Large Language Models against Harmful Fine-tuningcited by 72
  10. 2024A safety realignment framework via subspace-oriented model fusion for large language modelscited by 41
  11. 2024Immunization against harmful fine-tuning attackscited by 35
  12. 2024Safety Arithmetic: A Framework for Test-time Safety Alignment of Language Models by Steering Parameters and Activationscited by 32
38 more
  1. 2025MTSA: Multi-turn Safety Alignment for LLMs through Multi-round Red-teamingcited by 32
  2. 2024No Two Devils Alike: Unveiling Distinct Mechanisms of Fine-tuning Attackscited by 29
  3. 2025Understanding Refusal in Language Models with Sparse Autoencoderscited by 29
  4. 2024Holistic Automated Red Teaming for Large Language Models through Top-Down Test Case Generation and Multi-turn Interactioncited by 25
  5. 2025SafeMERGE: Preserving Safety Alignment in Fine-Tuned Large Language Models via Selective Layer-Wise Model Mergingcited by 25
  6. 2024JailbreakLens: Interpreting Jailbreak Mechanism in the Lens of Representation and Circuitcited by 22
  7. 2024Multi-Task Model Merging via Adaptive Weight Disentanglementcited by 22
  8. 2025NeuroStrike: Neuron-Level Attacks on Aligned LLMscited by 22
  9. 2024AlignGuard: Scalable Safety Alignment for Text-to-Image Generationcited by 20
  10. 2024Separate the Wheat from the Chaff: A Post-Hoc Approach to Safety Re-Alignment for Fine-Tuned Language Modelscited by 20
  11. 2024Have You Merged My Model? On The Robustness of Large Language Model IP Protection Methods Against Model Mergingcited by 19
  12. 2024Towards Inference-time Category-wise Safety Steering for Large Language Modelscited by 18
  13. 2025Safe Delta: Consistently Preserving Safety when Fine-Tuning LLMs on Diverse Datasetscited by 18
  14. 2024Towards Comprehensive Post Safety Alignment of Large Language Models via Safety Patchingcited by 17
  15. 2024WalledEval: A Comprehensive Safety Evaluation Toolkit for Large Language Modelscited by 17
  16. 2024Mix Data or Merge Models? Optimizing for Diverse Multi-Task Learningcited by 17
  17. 2024Task Arithmetic can Mitigate Synthetic-to-Real Gap in Automatic Speech Recognitioncited by 14
  18. 2024Unveiling Entity-Level Unlearning for Large Language Models: A Comprehensive Analysiscited by 14
  19. 2024Superficial Safety Alignment Hypothesiscited by 14
  20. 2024Data Advisor: Dynamic Data Curation for Safety Alignment of Large Language Modelscited by 13
  21. 2025Improving Consistency in Large Language Models through Chain of Guidancecited by 13
  22. 2025An Embarrassingly Simple Defense Against LLM Abliteration Attackscited by 13
  23. 2024Safeguard Fine-Tuned LLMs Through Pre- and Post-Tuning Model Mergingcited by 11
  24. 2025RainbowPlus: Enhancing Adversarial Prompt Generation via Evolutionary Quality-Diversity Searchcited by 11
  25. 2025GateBreaker: Gate-Guided Attacks on Mixture-of-Expert LLMscited by 11
  26. 2025Soteria: Language-Specific Functional Parameter Steering for Multilingual Safety Alignmentcited by 10
  27. 2025SafeSteer: Interpretable Safety Steering with Refusal-Evasion in LLMscited by 10
  28. 2026SafeNeuron: Neuron-Level Safety Alignment for Large Language Modelscited by 10
  29. 2024Toward Secure Tuning: Mitigating Security Risks from Instruction Fine-Tuningcited by 8
  30. 2025Three Minds, One Legend: Jailbreak Large Reasoning Model with Adaptive Stacked Cipherscited by 8
  31. 2025Defending MoE LLMs against Harmful Fine-Tuning via Safety Routing Alignmentcited by 8
  32. 2025Tree-based Dialogue Reinforced Policy Optimization for Red-Teaming Attackscited by 8
  33. 2026LSSF: Safety Alignment for Large Language Models through Low-Rank Safety Subspace Fusioncited by 8
  34. 2025The Rise of Darkness: Safety-Utility Trade-Offs in Role-Playing Dialogue Agentscited by 7
  35. 2025The Geometry of Harmfulness in LLMs through Subconcept Probingcited by 7
  36. 2025Anchoring Refusal Direction: Mitigating Safety Risks in Tuning via Projection Constraintcited by 7
  37. 2026Understanding and Preserving Safety in Fine-Tuned LLMscited by 7
  38. 2024Combining Domain and Alignment Vectors to Achieve Better Knowledge-Safety Trade-offs in LLMscited by 6

50 most cited of 112 All 112 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Safety at One Shot: Patching Fine-Tuned LLMs with A Single Instance
  2. 2026Can Safety Emerge from Weak Supervision? A Systematic Analysis of Small Language Models
  3. 2025Picky LLMs and Unreliable RMs: An Empirical Study on Safety Alignment after Instruction Tuning
  4. 2024LongSafety: Enhance Safety for Long-Context LLMs
  5. 2025Safety Alignment Depth in Large Language Models: A Markov Chain Perspective
  6. 2026Layer-wise Swapping for Generalizable Multilingual Safety
  7. 2024Multitask Mayhem: Unveiling and Mitigating Safety Gaps in LLMs Fine-tuning
  8. 2026Aligned but Fragile: Enhancing LLM Safety Robustness via Zeroth-Order Optimization