mlsanityData to 7 Oct 2026

On Cross-Layer Alignment for Model Fusion of Heterogeneous Neural Networks

arXiv 2110.15538 ↗First posted 29 Oct 2021cs.LG cs.CVcited by 9

Not on the map. Why?

Abstract (arXiv)

Layer-wise model fusion via optimal transport, named OTFusion, applies soft neuron association for unifying different pre-trained networks to save computational resources. While enjoying its success, OTFusion requires the input networks to have the same number of layers. To address this issue, we propose a novel model fusion framework, named CLAFusion, to fuse neural networks with a different number of layers, which we refer to as heterogeneous neural networks, via cross-layer alignment. The cross-layer alignment problem, which is an unbalanced assignment problem, can be solved efficiently using dynamic programming. Based on the cross-layer alignment, our framework balances the number of layers of neural networks before applying layer-wise model fusion. Our experiments indicate that CLAFusion, with an extra finetuning process, improves the accuracy of residual networks on the CIFAR10, CIFAR100, and Tiny-ImageNet datasets. Furthermore, we explore its practical usage for model compression and knowledge distillation when applying to the teacher-student setting.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Builds on 20 of its 45 references that are in the map. First posted on arXiv 29 Oct 2021.

Builds on

20 of 45 references.

References · 20

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  4. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  5. 2013Visualizing and Understanding Convolutional Networkscited by 2,278
  6. 2019Similarity of Neural Network Representations Revisitedcited by 1,695
  7. 2020Knowledge Distillation: A Surveycited by 1,578
  8. 2016Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfercited by 1,059
  9. 2020Federated Learning with Matched Averagingcited by 564
  10. 2019Similarity-Preserving Knowledge Distillationcited by 507
  11. 2017A Downsampled Variant of ImageNet as an Alternative to the CIFAR datasetscited by 374
  12. 2015Net2Net: Accelerating Learning via Knowledge Transfercited by 245
8 more
  1. 2019Model Fusion via Optimal Transportcited by 217
  2. 2013Fast Computation of Wasserstein Barycenterscited by 216
  3. 2015Convergent Learning: Do different neural networks learn the same representations?cited by 215
  4. 2020Cross-Layer Distillation with Semantic Calibrationcited by 141
  5. 2019Are All Layers Created Equal?cited by 94
  6. 2016Network Morphismcited by 63
  7. 2018Unifying and Merging Well-trained Deep Neural Networks for Inference Stagecited by 8
  8. 2018Non-Iterative Knowledge Fusion in Deep Convolutional Neural Networkscited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 3; 2025: 3; 2026: 1.

9 papers.

  1. 2024Model Merging in LLMs, MLLMs, and Beyond: Methods, Theories, Applications and Opportunitiescited by 247
  2. 2023Transformer Fusion with Optimal Transportcited by 35
  3. 2024Training-Free Pretrained Model Mergingcited by 32
  4. 2023Towards Personalized Federated Learning via Heterogeneous Model Reassemblycited by 31
  5. 2025Sens-Merging: Sensitivity-Guided Parameter Balancing for Merging Large Language Modelscited by 16
  6. 2024Applications of Knowledge Distillation in Remote Sensing: A Surveycited by 8
  7. 2025SE-Merging: A Self-Enhanced Approach for Dynamic Model Mergingcited by 1
  8. 2026Partial Fusion of Neural Networks: Efficient Tradeoffs Between Ensembles and Weight Aggregationcited by 1
  9. 2025Fusion of Graph Neural Networks via Optimal Transportcited by 0