# C2M3: Cycle-Consistent Multi-Model Merging

arXiv: https://arxiv.org/abs/2405.17897 · HTML: https://mlsanity.com/paper/2405.17897/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 42 references that are in the map.
- First posted on arXiv 28 May 2024.

## Builds on (most cited first, 23 of 23)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Similarity of Neural Network Representations Revisited](https://mlsanity.com/paper/1905.00414/) (2019)
- [Linear Mode Connectivity and the Lottery Ticket Hypothesis](https://mlsanity.com/paper/1912.05671/) (2019)
- [Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNs](https://mlsanity.com/paper/1802.10026/) (2018)
- [Git Re-Basin: Merging Models modulo Permutation Symmetries](https://mlsanity.com/paper/2209.04836/) (2022)
- [Flower: A Friendly Federated Learning Research Framework](https://mlsanity.com/paper/2007.14390/) (2020)
- [Dataless Knowledge Fusion by Merging Weights of Language Models](https://mlsanity.com/paper/2212.09849/) (2022)
- [Essentially No Barriers in Neural Network Energy Landscape](https://mlsanity.com/paper/1803.00885/) (2018)
- [The Role of Permutation Invariance in Linear Mode Connectivity of Neural Networks](https://mlsanity.com/paper/2110.06296/) (2021)
- [Model Fusion via Optimal Transport](https://mlsanity.com/paper/1910.05653/) (2019)
- [EMNIST: an extension of MNIST to handwritten letters](https://mlsanity.com/paper/1702.05373/) (2017)
- [ZipIt! Merging Models from Different Tasks without Training](https://mlsanity.com/paper/2305.03053/) (2023)
- [REPAIR: REnormalizing Permuted Activations for Interpolation Repair](https://mlsanity.com/paper/2211.08403/) (2022)
- [Neural Graph Matching Network: Learning Lawler's Quadratic Assignment Problem with Extension to Hypergraph and Multiple-graph Matching](https://mlsanity.com/paper/1911.11308/) (2019)
- [Convergence Rate of Frank-Wolfe for Non-Convex Objectives](https://mlsanity.com/paper/1607.00345/) (2016)
- [Re-basin via implicit Sinkhorn differentiation](https://mlsanity.com/paper/2212.12042/) (2022)
- [Mechanistic Mode Connectivity](https://mlsanity.com/paper/2211.08422/) (2022)
- [Going Beyond Linear Mode Connectivity: The Layerwise Linear Feature Connectivity](https://mlsanity.com/paper/2307.08286/) (2023)
- [Higher-order Projected Power Iterations for Scalable Multi-Matching](https://mlsanity.com/paper/1811.10541/) (2018)
- [Sparse Quadratic Optimisation over the Stiefel Manifold with Application to Permutation Synchronisation](https://mlsanity.com/paper/2110.00053/) (2021)

## Built on by (most cited first, 20 of 20)

- [Model Merging in LLMs, MLLMs, and Beyond: Methods, Theories, Applications and Opportunities](https://mlsanity.com/paper/2408.07666/) (2024)
- [Task Singular Vectors: Reducing Task Interference in Model Merging](https://mlsanity.com/paper/2412.00081/) (2024)
- [Modeling Multi-Task Model Merging as Adaptive Projective Gradient Descent](https://mlsanity.com/paper/2501.01230/) (2025)
- [Update Your Transformer to the Latest Release: Re-Basin of Task Vectors](https://mlsanity.com/paper/2505.22697/) (2025)
- [Generalized Linear Mode Connectivity for Transformers](https://mlsanity.com/paper/2506.22712/) (2025)
- [ATM: Improving Model Merging by Alternating Tuning and Merging](https://mlsanity.com/paper/2411.03055/) (2024)
- [MERGE3: Efficient Evolutionary Merging on Consumer-grade GPUs](https://mlsanity.com/paper/2502.10436/) (2025)
- [On Task Vectors and Gradients](https://mlsanity.com/paper/2508.16082/) (2025)
- [Demystifying Mergeability: Interpretable Properties to Predict Model Merging Success](https://mlsanity.com/paper/2601.22285/) (2026)
- [Model Merging in the Essential Subspace](https://mlsanity.com/paper/2602.20208/) (2026)
- [MASS: MoErging through Adaptive Subspace Selection](https://mlsanity.com/paper/2504.05342/) (2025)
- [Model Merging Improves Zero-Shot Generalization in Bioacoustic Foundation Models](https://mlsanity.com/paper/2511.05171/) (2025)
- [Preference-Aligned LoRA Merging: Preserving Subspace Coverage and Addressing Directional Anisotropy](https://mlsanity.com/paper/2603.26299/) (2026)
- [Enough is as good as a feast: A Comprehensive Analysis of How Reinforcement Learning Mitigates Task Conflicts in LLMs](https://mlsanity.com/paper/2607.22039/) (2026)
- [StatsMerging: Statistics-Guided Model Merging via Task-Specific Teacher Distillation](https://mlsanity.com/paper/2506.04567/) (2025)
- [Hyper Diffusion Avatars: Dynamic Human Avatar Generation using Network Weight Space Diffusion](https://mlsanity.com/paper/2509.04145/) (2025)
- [AlignMerge - Alignment-Preserving Large Language Model Merging via Fisher-Guided Geometric Constraints](https://mlsanity.com/paper/2512.16245/) (2025)
- [TwistedMerge: Certified Higher-Order Diagnostics and Abstention for Model Merging](https://mlsanity.com/paper/2607.20887/) (2026)
- [Optimizer-Induced Mode Connectivity: From AdamW to Muon](https://mlsanity.com/paper/2605.09991/) (2026)
- [Beyond Structural Symmetries: Linear Mode Connectivity via Neuron Identifiability](https://mlsanity.com/paper/2606.04754/) (2026)

## Abstract (arXiv)

In this paper, we present a novel data-free method for merging neural networks in weight space. Differently from most existing works, our method optimizes for the permutations of network neurons globally across all layers. This allows us to enforce cycle consistency of the permutations when merging N ≥ 3 models, allowing circular compositions of permutations to be computed without accumulating error along the path. We qualitatively and quantitatively motivate the need for such a constraint, showing its benefits when merging sets of models in scenarios spanning varying architectures and datasets. We finally show that, when coupled with activation renormalization, our approach yields the best results in the task.