mlsanityData to 14 Oct 2026

C2M3: Cycle-Consistent Multi-Model Merging

arXiv 2405.17897 ↗First posted 28 May 2024cs.LGcited by 20

Not on the map. Why?

Abstract (arXiv)

In this paper, we present a novel data-free method for merging neural networks in weight space. Differently from most existing works, our method optimizes for the permutations of network neurons globally across all layers. This allows us to enforce cycle consistency of the permutations when merging N ≥ 3 models, allowing circular compositions of permutations to be computed without accumulating error along the path. We qualitatively and quantitatively motivate the need for such a constraint, showing its benefits when merging sets of models in scenarios spanning varying architectures and datasets. We finally show that, when coupled with activation renormalization, our approach yields the best results in the task.

Key facts. Cited by 20 papers in the map (data through 14 Oct 2026). Builds on 23 of its 42 references that are in the map. First posted on arXiv 28 May 2024.

Builds on

23 of 42 references.

References · 23

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  3. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 7,005
  4. 2016Layer Normalizationcited by 5,669
  5. 2019Similarity of Neural Network Representations Revisitedcited by 1,755
  6. 2019Linear Mode Connectivity and the Lottery Ticket Hypothesiscited by 637
  7. 2018Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNscited by 588
  8. 2022Git Re-Basin: Merging Models modulo Permutation Symmetriescited by 471
  9. 2020Flower: A Friendly Federated Learning Research Frameworkcited by 455
  10. 2022Dataless Knowledge Fusion by Merging Weights of Language Modelscited by 387
  11. 2018Essentially No Barriers in Neural Network Energy Landscapecited by 330
  12. 2021The Role of Permutation Invariance in Linear Mode Connectivity of Neural Networkscited by 291
11 more
  1. 2019Model Fusion via Optimal Transportcited by 219
  2. 2017EMNIST: an extension of MNIST to handwritten letterscited by 213
  3. 2023ZipIt! Merging Models from Different Tasks without Trainingcited by 179
  4. 2022REPAIR: REnormalizing Permuted Activations for Interpolation Repaircited by 120
  5. 2019Neural Graph Matching Network: Learning Lawler's Quadratic Assignment Problem with Extension to Hypergraph and Multiple-graph Matchingcited by 84
  6. 2016Convergence Rate of Frank-Wolfe for Non-Convex Objectivescited by 63
  7. 2022Re-basin via implicit Sinkhorn differentiationcited by 62
  8. 2022Mechanistic Mode Connectivitycited by 50
  9. 2023Going Beyond Linear Mode Connectivity: The Layerwise Linear Feature Connectivitycited by 50
  10. 2018Higher-order Projected Power Iterations for Scalable Multi-Matchingcited by 24
  11. 2021Sparse Quadratic Optimisation over the Stiefel Manifold with Application to Permutation Synchronisationcited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 3; 2025: 10; 2026: 7.

20 papers.

Sort:

By area:

  1. 2024Model Merging in LLMs, MLLMs, and Beyond: Methods, Theories, Applications and Opportunitiescited by 254
  2. 2024Task Singular Vectors: Reducing Task Interference in Model Mergingcited by 147
  3. 2025Modeling Multi-Task Model Merging as Adaptive Projective Gradient Descentcited by 39
  4. 2025Update Your Transformer to the Latest Release: Re-Basin of Task Vectorscited by 17
  5. 2025Generalized Linear Mode Connectivity for Transformerscited by 16
  6. 2024ATM: Improving Model Merging by Alternating Tuning and Mergingcited by 11
  7. 2025MERGE3: Efficient Evolutionary Merging on Consumer-grade GPUscited by 10
  8. 2025On Task Vectors and Gradientscited by 7
  9. 2026Demystifying Mergeability: Interpretable Properties to Predict Model Merging Successcited by 7
  10. 2026Model Merging in the Essential Subspacecited by 7
  11. 2025MASS: MoErging through Adaptive Subspace Selectioncited by 5
  12. 2025Model Merging Improves Zero-Shot Generalization in Bioacoustic Foundation Modelscited by 2
8 more
  1. 2026Preference-Aligned LoRA Merging: Preserving Subspace Coverage and Addressing Directional Anisotropycited by 2
  2. 2026Enough is as good as a feast: A Comprehensive Analysis of How Reinforcement Learning Mitigates Task Conflicts in LLMscited by 2
  3. 2025StatsMerging: Statistics-Guided Model Merging via Task-Specific Teacher Distillationcited by 1
  4. 2025Hyper Diffusion Avatars: Dynamic Human Avatar Generation using Network Weight Space Diffusioncited by 1
  5. 2025AlignMerge - Alignment-Preserving Large Language Model Merging via Fisher-Guided Geometric Constraintscited by 1
  6. 2026TwistedMerge: Certified Higher-Order Diagnostics and Abstention for Model Mergingcited by 1
  7. 2026Optimizer-Induced Mode Connectivity: From AdamW to Muoncited by 0
  8. 2026Beyond Structural Symmetries: Linear Mode Connectivity via Neuron Identifiabilitycited by 0