# Modularized Morphing of Neural Networks

arXiv: https://arxiv.org/abs/1701.03281 · HTML: https://mlsanity.com/paper/1701.03281/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 Jan 2017.

## Built on by (most cited first, 5 of 5)

- [Automated Progressive Learning for Efficient Training of Vision Transformers](https://mlsanity.com/paper/2203.14509/) (2022)
- [DyRep: Bootstrapping Training with Dynamic Re-parameterization](https://mlsanity.com/paper/2203.12868/) (2022)
- [Stacked BNAS: Rethinking Broad Convolutional Neural Network for Neural Architecture Search](https://mlsanity.com/paper/2111.07722/) (2021)
- [Reparameterization through Spatial Gradient Scaling](https://mlsanity.com/paper/2303.02733/) (2023)
- [Ever Evolving Evaluator (EV3): Towards Flexible and Reliable Meta-Optimization for Knowledge Distillation](https://mlsanity.com/paper/2310.18893/) (2023)

## Abstract (arXiv)

In this work we study the problem of network morphism, an effective learning scheme to morph a well-trained neural network to a new one with the network function completely preserved. Different from existing work where basic morphing types on the layer level were addressed, we target at the central problem of network morphism at a higher level, i.e., how a convolutional layer can be morphed into an arbitrary module of a neural network. To simplify the representation of a network, we abstract a module as a graph with blobs as vertices and convolutional layers as edges, based on which the morphing process is able to be formulated as a graph transformation problem. Two atomic morphing operations are introduced to compose the graphs, based on which modules are classified into two families, i.e., simple morphable modules and complex modules. We present practical morphing solutions for both of these two families, and prove that any reasonable module can be morphed from a single convolutional layer. Extensive experiments have been conducted based on the state-of-the-art ResNet on benchmark datasets, and the effectiveness of the proposed solution has been verified.