# M-Loss: Quantifying Model Merging Compatibility with Limited Unlabeled Data

arXiv: https://arxiv.org/abs/2602.08564 · HTML: https://mlsanity.com/paper/2602.08564/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 46 references that are in the map.
- First posted on arXiv 9 Feb 2026.

## Builds on (most cited first, 23 of 23)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Gaussian Error Linear Units (GELUs)](https://mlsanity.com/paper/1606.08415/) (2016)
- [Describing Textures in the Wild](https://mlsanity.com/paper/1311.3618/) (2013)
- [Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time](https://mlsanity.com/paper/2203.05482/) (2022)
- [On the Convergence of FedAvg on Non-IID Data](https://mlsanity.com/paper/1907.02189/) (2019)
- [TIES-Merging: Resolving Interference When Merging Models](https://mlsanity.com/paper/2306.01708/) (2023)
- [Remote Sensing Image Scene Classification: Benchmark and State of the Art](https://mlsanity.com/paper/1703.00121/) (2017)
- [Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunch](https://mlsanity.com/paper/2311.03099/) (2023)
- [Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNs](https://mlsanity.com/paper/1802.10026/) (2018)
- [FedBN: Federated Learning on Non-IID Features via Local Batch Normalization](https://mlsanity.com/paper/2102.07623/) (2021)
- [Dataless Knowledge Fusion by Merging Weights of Language Models](https://mlsanity.com/paper/2212.09849/) (2022)
- [BatchEnsemble: An Alternative Approach to Efficient Ensemble and Lifelong Learning](https://mlsanity.com/paper/2002.06715/) (2020)
- [Essentially No Barriers in Neural Network Energy Landscape](https://mlsanity.com/paper/1803.00885/) (2018)
- [Topology and Geometry of Half-Rectified Network Optimization](https://mlsanity.com/paper/1611.01540/) (2016)
- [The rising costs of training frontier AI models](https://mlsanity.com/paper/2405.21015/) (2024)
- [Going Beyond Linear Mode Connectivity: The Layerwise Linear Feature Connectivity](https://mlsanity.com/paper/2307.08286/) (2023)
- [vTrain: A Simulation Framework for Evaluating Cost-effective and Compute-optimal Large Language Model Training](https://mlsanity.com/paper/2312.12391/) (2023)
- [Training-Free Model Merging for Multi-target Domain Adaptation](https://mlsanity.com/paper/2407.13771/) (2024)
- [Generalizability of Machine Learning Models: Quantitative Evaluation of Three Methodological Pitfalls](https://mlsanity.com/paper/2202.01337/) (2022)
- [Deep Time Series Models for Scarce Data](https://mlsanity.com/paper/2103.09348/) (2021)
- [Leveraging Submodule Linearity Enhances Task Arithmetic Performance in LLMs](https://mlsanity.com/paper/2504.10902/) (2025)
- [Reducing the Barriers to Entry for Foundation Model Training](https://mlsanity.com/paper/2404.08811/) (2024)
- [More Compute Is What You Need](https://mlsanity.com/paper/2404.19484/) (2024)

## Built on by (most cited first, 2 of 2)

- [MPU: Towards Secure and Privacy-Preserving Knowledge Unlearning for Large Language Models](https://mlsanity.com/paper/2602.23798/) (2026)
- [More Data, Worse Decisions? Preference Reversals in Neural Networks under Gram Incompatibility](https://mlsanity.com/paper/2607.27255/) (2026)

## Abstract (arXiv)

Training of large-scale models is both computationally intensive and often constrained by the availability of labeled data. Model merging offers a compelling alternative by directly integrating the weights of multiple source models without requiring additional data or extensive training. However, conventional model merging techniques, such as parameter averaging, often suffer from the unintended combination of non-generalizable features, especially when source models exhibit significant weight disparities. Comparatively, model ensembling generally provides more stable and superior performance that aggregates multiple models by averaging outputs. However, it incurs higher inference costs and increased storage requirements. While previous studies experimentally showed the similarities between model merging and ensembling, theoretical evidence and evaluation metrics remain lacking. To address this gap, we introduce Merging-ensembling loss (M-Loss), a novel evaluation metric that quantifies the compatibility of merging source models using very limited unlabeled data. By measuring the discrepancy between parameter averaging and model ensembling at layer and node levels, M-Loss facilitates more effective merging strategies. Specifically, M-Loss serves both as a quantitative criterion of the theoretical feasibility of model merging, and a guide for parameter significance in model pruning. Our theoretical analysis and empirical evaluations demonstrate that incorporating M-Loss into the merging process significantly improves the alignment between merged models and model ensembling, providing a scalable and efficient framework for accurate model consolidation.