# Sens-Merging: Sensitivity-Guided Parameter Balancing for Merging Large Language Models

arXiv: https://arxiv.org/abs/2502.12420 · HTML: https://mlsanity.com/paper/2502.12420/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 28 references that are in the map.
- First posted on arXiv 18 Feb 2025.

## Builds on (most cited first, 24 of 24)

- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Measuring Mathematical Problem Solving With the MATH Dataset](https://mlsanity.com/paper/2103.03874/) (2021)
- [HellaSwag: Can a Machine Really Finish Your Sentence?](https://mlsanity.com/paper/1905.07830/) (2019)
- [Program Synthesis with Large Language Models](https://mlsanity.com/paper/2108.07732/) (2021)
- [TruthfulQA: Measuring How Models Mimic Human Falsehoods](https://mlsanity.com/paper/2109.07958/) (2021)
- [TIES-Merging: Resolving Interference When Merging Models](https://mlsanity.com/paper/2306.01708/) (2023)
- [Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunch](https://mlsanity.com/paper/2311.03099/) (2023)
- [Linear Mode Connectivity and the Lottery Ticket Hypothesis](https://mlsanity.com/paper/1912.05671/) (2019)
- [Merging Models with Fisher-Weighted Averaging](https://mlsanity.com/paper/2111.09832/) (2021)
- [Git Re-Basin: Merging Models modulo Permutation Symmetries](https://mlsanity.com/paper/2209.04836/) (2022)
- [Dataless Knowledge Fusion by Merging Weights of Language Models](https://mlsanity.com/paper/2212.09849/) (2022)
- [The Role of Permutation Invariance in Linear Mode Connectivity of Neural Networks](https://mlsanity.com/paper/2110.06296/) (2021)
- [Model Merging in LLMs, MLLMs, and Beyond: Methods, Theories, Applications and Opportunities](https://mlsanity.com/paper/2408.07666/) (2024)
- [Efficiently Identifying Task Groupings for Multi-Task Learning](https://mlsanity.com/paper/2109.04617/) (2021)
- [ZipIt! Merging Models from Different Tasks without Training](https://mlsanity.com/paper/2305.03053/) (2023)
- [REPAIR: REnormalizing Permuted Activations for Interpolation Repair](https://mlsanity.com/paper/2211.08403/) (2022)
- [Topology and Geometry of Half-Rectified Network Optimization](https://mlsanity.com/paper/1611.01540/) (2016)
- [Optimizing Mode Connectivity via Neuron Alignment](https://mlsanity.com/paper/2009.02439/) (2020)
- [Mitigating Catastrophic Forgetting in Language Transfer via Model Merging](https://mlsanity.com/paper/2407.08699/) (2024)
- [On Cross-Layer Alignment for Model Fusion of Heterogeneous Neural Networks](https://mlsanity.com/paper/2110.15538/) (2021)
- [GAN Cocktail: mixing GANs without dataset access](https://mlsanity.com/paper/2106.03847/) (2021)

## Built on by (most cited first, 16 of 16)

- [Unlocking Efficient Long-to-Short LLM Reasoning with Model Merging](https://mlsanity.com/paper/2503.20641/) (2025)
- [Don't Overthink It: A Survey of Efficient R1-style Large Reasoning Models](https://mlsanity.com/paper/2508.02120/) (2025)
- [Locate, Steer, and Improve: A Practical Survey of Actionable Mechanistic Interpretability in Large Language Models](https://mlsanity.com/paper/2601.14004/) (2026)
- [Activation-Guided Consensus Merging for Large Language Models](https://mlsanity.com/paper/2505.14009/) (2025)
- [MergePipe: A Budget-Aware Parameter Management System for Scalable LLM Merging](https://mlsanity.com/paper/2602.13273/) (2026)
- [RCP-Merging: Merging Long Chain-of-Thought Models with Domain-Specific Models by Considering Reasoning Capability as Prior](https://mlsanity.com/paper/2508.03140/) (2025)
- [Label-Free Cross-Task LoRA Merging with Null-Space Compression](https://mlsanity.com/paper/2603.26317/) (2026)
- [Preference-Aligned LoRA Merging: Preserving Subspace Coverage and Addressing Directional Anisotropy](https://mlsanity.com/paper/2603.26299/) (2026)
- [Access Sets Matter: Budgeting Expert Reads for Scalable Weight-Space Model Merging](https://mlsanity.com/paper/2605.29489/) (2026)
- [ReasonAny: Incorporating Reasoning Capability to Any Model via Simple and Effective Model Merging](https://mlsanity.com/paper/2601.05560/) (2026)
- [Merging Beyond: Streaming LLM Updates via Activation-Guided Rotations](https://mlsanity.com/paper/2602.03237/) (2026)
- [DiMtextsuperscript3: Bridging Multilingual and Multimodal Models via Direction- and Magnitude-Aware Merging](https://mlsanity.com/paper/2605.12960/) (2026)
- [ReCatcher: Towards LLMs Regression Testing for Code Generation](https://mlsanity.com/paper/2507.19390/) (2025)
- [Data-Free Layer-Adaptive Merging via Fisher Information for Long-to-Short Reasoning LLMs](https://mlsanity.com/paper/2603.21705/) (2026)
- [Scaling Mobile Agent Systems: From Capability Density to Collective Intelligence](https://mlsanity.com/paper/2605.08124/) (2026)
- [Towards Efficient Reasoning in LLM-Based Recommender Systems via Model Merging](https://mlsanity.com/paper/2608.10447/) (2026)

## Abstract (arXiv)

Recent advances in large language models have led to numerous task-specialized fine-tuned variants, creating a need for efficient model merging techniques that preserve specialized capabilities while avoiding costly retraining. While existing task vector-based merging methods show promise, they typically apply uniform coefficients across all parameters, overlooking varying parameter importance both within and across tasks. We present Sens-Merging, a sensitivity-guided coefficient adjustment method that enhances existing model merging techniques by operating at both task-specific and cross-task levels. Our method analyzes parameter sensitivity within individual tasks and evaluates cross-task transferability to determine optimal merging coefficients. Extensive experiments on Mistral 7B and LLaMA2-7B/13B models demonstrate that Sens-Merging significantly improves performance across general knowledge, mathematical reasoning, and code generation tasks. Notably, when combined with existing merging techniques, our method enables merged models to outperform specialized fine-tuned models, particularly in code generation tasks. Our findings reveal important trade-offs between task-specific and cross-task scalings, providing insights for future model merging strategies.