Optimal Brain Iterative Merging: Mitigating Interference in LLM Merging
Not on the map. Why?
Abstract (arXiv)
Large Language Models (LLMs) have demonstrated impressive capabilities, but their high computational costs pose challenges for customization. Model merging offers a cost-effective alternative, yet existing methods suffer from interference among parameters, leading to performance degradation. In this work, we propose Optimal Brain Iterative Merging (OBIM), a novel method designed to mitigate both intra-model and inter-model interference. OBIM consists of two key components: (1) A saliency measurement mechanism that evaluates parameter importance based on loss changes induced by individual weight alterations, reducing intra-model interference by preserving only high-saliency parameters. (2) A mutually exclusive iterative merging framework, which incrementally integrates models using a binary mask to avoid direct parameter averaging, thereby mitigating inter-model interference. We validate OBIM through experiments on both Supervised Fine-Tuned (SFT) models and post-pretrained checkpoints. The results show that OBIM significantly outperforms existing merging techniques. Overall, OBIM provides an effective and practical solution for enhancing LLM merging.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 27 of its 47 references that are in the map. First posted on arXiv 17 Feb 2025.
Builds on
27 of 47 references.
References · 27
- 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
- 2021Training Verifiers to Solve Math Word Problemscited by 10,109
- 2021Evaluating Large Language Models Trained on Codecited by 8,320
- 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
- 2021Program Synthesis with Large Language Modelscited by 3,637
- 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
- 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,396
- 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,125
- 2023C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Modelscited by 708
- 2023Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunchcited by 694
- 2023WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instructcited by 625
- 2021Merging Models with Fisher-Weighted Averagingcited by 621
15 more
- 2024DataComp-LM: In search of the next generation of training sets for language modelscited by 398
- 2022Dataless Knowledge Fusion by Merging Weights of Language Modelscited by 387
- 2022Optimal Brain Compression: A Framework for Accurate Post-Training Quantization and Pruningcited by 322
- 2024Evolutionary Optimization of Model Merging Recipescited by 208
- 2023Model Breadcrumbs: Scaling Multi-Task Model Merging with Sparse Maskscited by 133
- 2024Localizing Task Information for Improved Model Merging and Compressioncited by 131
- 2024Model Stock: All we need is just a few fine-tuned modelscited by 92
- 2024DELLA-Merging: Reducing Interference in Model Merging through Magnitude-Based Samplingcited by 81
- 2024FuseChat: Knowledge Fusion of Chat Modelscited by 55
- 2023Controlled Text Generation via Language Model Arithmeticcited by 51
- 2023Separate the Wheat from the Chaff: Model Deficiency Unlearning via Parameter-Efficient Module Operationcited by 33
- 2024Checkpoint Merging via Bayesian Optimization in LLM Pretrainingcited by 27
- 2023Robust Weight Signatures: Gaining Robustness as Easy as Patching Weights?cited by 13
- 2024Beyond Task Vectors: Selective Task Arithmetic Based on Importance Metricscited by 11
- 2024DogeRM: Equipping Reward Models with Domain Knowledge through Model Mergingcited by 5
Built on by
Citing papers per year: 2025: 1; 2026: 1.
2 papers.