mlsanityData to 14 Oct 2026

Combining Domain and Alignment Vectors to Achieve Better Knowledge-Safety Trade-offs in LLMs

arXiv 2411.06824 ↗First posted 11 Nov 2024cs.AIcited by 6

Not on the map. Why?

Abstract (arXiv)

There is a growing interest in training domain-expert LLMs that excel in specific technical fields compared to their general-purpose instruction-tuned counterparts. However, these expert models often experience a loss in their safety abilities in the process, making them capable of generating harmful content. As a solution, we introduce an efficient and effective merging-based alignment method called textscMergeAlign that interpolates the domain and alignment vectors, creating safer domain-specific models while preserving their utility. We apply textscMergeAlign on Llama3 variants that are experts in medicine and finance, obtaining substantial alignment improvements with minimal to no degradation on domain-specific benchmarks. We study the impact of model merging through model similarity metrics and contributions of individual models being merged. We hope our findings open new research avenues and inspire more efficient development of safe expert LLMs.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Builds on 23 of its 29 references that are in the map. First posted on arXiv 11 Nov 2024.

Builds on

23 of 29 references.

References · 23

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  2. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  4. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  5. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  6. 2021Program Synthesis with Large Language Modelscited by 3,637
  7. 2023A Survey of Large Language Modelscited by 2,576
  8. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,396
  9. 2024Qwen2.5-Coder Technical Reportcited by 1,203
  10. 2022Editing Models with Task Arithmeticcited by 1,120
  11. 2024Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvementcited by 1,090
  12. 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Datasetcited by 860
11 more
  1. 2023TIES-Merging: Resolving Interference When Merging Modelscited by 818
  2. 2023Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunchcited by 694
  3. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedbackcited by 656
  4. 2024ORPO: Monolithic Preference Optimization without Reference Modelcited by 562
  5. 2023MAmmoTH: Building Math Generalist Models through Hybrid Instruction Tuningcited by 487
  6. 2024SALAD-Bench: A Hierarchical and Comprehensive Safety Benchmark for Large Language Modelscited by 228
  7. 2023Mitigating the Alignment Tax of RLHFcited by 183
  8. 2023Domain Specialization as the Key to Make Large Language Models Disruptive: A Comprehensive Surveycited by 135
  9. 2024Language Models are Homer Simpson! Safety Re-Alignment of Fine-tuned Language Models through Task Arithmeticcited by 112
  10. 2024Instruction Pre-Training: Language Models are Supervised Multitask Learnerscited by 63
  11. 2024A Deep Dive into the Trade-Offs of Parameter-Efficient Preference Alignment Techniquescited by 16

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 2; 2025: 3; 2026: 1.

6 papers.

  1. 2024Position: Theory of Mind Benchmarks are Broken for Large Language Modelscited by 31
  2. 2024Enabling Realtime Reinforcement Learning at Scale with Staggered Asynchronous Inferencecited by 7
  3. 2025LED-Merging: Mitigating Safety-Utility Conflicts in Model Merging with Location-Election-Disjointcited by 7
  4. 2025Adaptive Helpfulness-Harmlessness Alignment with Preference Vectorscited by 6
  5. 2025MGC: A Compiler Framework Exploiting Compositional Blindness in Aligned LLMs for Malware Generationcited by 0
  6. 2026Domain Arithmetic: One-Shot VLA Adaptation under Environmental Shiftscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Model Merging and Safety Alignment: One Bad Model Spoils the Bunch
  2. 2026The Art of (Mis)alignment: How Fine-Tuning Methods Effectively Misalign and Realign LLMs in Post-Training
  3. 2024Unconstrained Model Merging for Enhanced LLM Reasoning
  4. 2024Keeping LLMs Aligned After Fine-tuning: The Crucial Role of Prompt Templates
  5. 2024H3Fusion: Helpful, Harmless, Honest Fusion of Aligned LLMs
  6. 2025A Comprehensive Evaluation framework of Alignment Techniques for LLMs
  7. 2024Systematic Evaluation of LLM-as-a-Judge in LLM Alignment Tasks: Explainable Metrics and Diverse Prompt Templates
  8. 2025Merging Continual Pretraining Models for Domain-Specialized LLMs: A Case Study in Finance