mlsanityData to 7 Oct 2026

Network Morphism

arXiv 1603.01670 ↗First posted 5 Mar 2016cs.LG cs.CV cs.NEcited by 63

Not on the map. Why?

Abstract (arXiv)

We present in this paper a systematic study on how to morph a well-trained neural network to a new one so that its network function can be completely preserved. We define this as network morphism in this research. After morphing a parent network, the child network is expected to inherit the knowledge from its parent network and also has the potential to continue growing into a more powerful one with much shortened training time. The first requirement for this network morphism is its ability to handle diverse morphing types of networks, including changes of depth, width, kernel size, and even subnet. To meet this requirement, we first introduce the network morphism equations, and then develop novel morphing algorithms for all these morphing types for both classic and convolutional neural networks. The second requirement for this network morphism is its ability to deal with non-linearity in a network. We propose a family of parametric-activation functions to facilitate the morphing of any continuous non-linear activation neurons. Experimental results on benchmark datasets and typical neural networks demonstrate the effectiveness of the proposed network morphism scheme.

Key facts. Cited by 63 papers in the map (data through 7 Oct 2026). First posted on arXiv 5 Mar 2016.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 12; 2022: 15; 2023: 10; 2024: 9; 2025: 3; 2026: 14.

63 papers.

Sort:

By area:

  1. 2023Neural Architecture Search: Insights from 1000 Paperscited by 136
  2. 2023Learning to Grow Pretrained Models for Efficient Transformer Trainingcited by 81
  3. 2022GradMax: Growing Neural Networks using Gradient Informationcited by 64
  4. 2022Staged Training for Transformer Language Modelscited by 49
  5. 2021Firefly Neural Architecture Descent: a General Approach for Growing Neural Networkscited by 48
  6. 2022EfficientTrain: Exploring Generalized Curriculum Learning for Training Visual Backbonescited by 42
  7. 2022Automated Progressive Learning for Efficient Training of Vision Transformerscited by 36
  8. 2023Masked Structural Growth for 2x Faster Language Model Pre-trainingcited by 32
  9. 2021The Elastic Lottery Ticket Hypothesiscited by 23
  10. 2022When, where, and how to add new neurons to ANNscited by 18
  11. 2022Efficient Architecture Search for Diverse Taskscited by 18
  12. 2023Seeking Neural Nuggets: Knowledge Transfer in Large Language Models from a Parametric Perspectivecited by 17
38 more
  1. 2024EfficientTrain++: Generalized Curriculum Learning for Efficient Visual Backbone Trainingcited by 16
  2. 2021GROWN: GRow Only When Necessary for Continual Learningcited by 11
  3. 2022Neural Architecture Search for Dense Prediction Tasks in Computer Visioncited by 11
  4. 2022DyRep: Bootstrapping Training with Dynamic Re-parameterizationcited by 11
  5. 2022Progressive Transformation Learning for Leveraging Virtual Images in Trainingcited by 10
  6. 2021On Cross-Layer Alignment for Model Fusion of Heterogeneous Neural Networkscited by 9
  7. 2024NACHOS: Neural Architecture Search for Hardware Constrained Early Exit Neural Networkscited by 9
  8. 2022PRE-NAS: Predictor-assisted Evolutionary Neural Architecture Searchcited by 8
  9. 2022Equivariance-aware Architectural Optimization of Neural Networkscited by 8
  10. 2023Accelerated Training via Incrementally Growing Neural Networks using Variance Transfer and Learning Rate Adaptationcited by 8
  11. 2021Beyond Fine-tuning: Classifying High Resolution Mammograms using Function-Preserving Transformationscited by 5
  12. 2021Manas: Mining Software Repositories to Assist AutoMLcited by 5
  13. 2022A Classification of G-invariant Shallow Neural Networkscited by 5
  14. 2022Quant 4.0: Engineering Quantitative Investment with Automated, Explainable and Knowledge-driven Artificial Intelligencecited by 5
  15. 2025LESA: Learnable LLM Layer Scaling-Upcited by 5
  16. 2021Input Invex Neural Networkcited by 4
  17. 2022Neural Architecture Search using Property Guided Synthesiscited by 4
  18. 2023Deep Fusion: Efficient Network Training via Pre-trained Initializationscited by 4
  19. 2024Landscape-Aware Growing: The Power of a Little LAGcited by 4
  20. 2024Efficient Training of Large Vision Models via Advanced Automated Progressive Learningcited by 4
  21. 2024Neural Metamorphosiscited by 4
  22. 2025Optimal Transport Depth Up-Scalingcited by 4
  23. 2021Stacked BNAS: Rethinking Broad Convolutional Neural Network for Neural Architecture Searchcited by 3
  24. 2024When To Grow? A Fitting Risk-Aware Policy for Layer Growing in Deep Neural Networkscited by 3
  25. 2021A Novel Framework for Neural Architecture Search in the Hill Climbing Domaincited by 2
  26. 2021Efficient Visual Recognition with Deep Neural Networks: A Survey on Recent Advances and New Directionscited by 2
  27. 2022Differentiable Search of Accurate and Robust Architecturescited by 2
  28. 2024Multi-Objective Evolutionary Neural Architecture Search for Recurrent Neural Networkscited by 2
  29. 2021Differentiable Neural Architecture Search with Morphism-based Transformable Backbone Architecturescited by 1
  30. 2023Efficient Latency-Aware CNN Depth Compression via Two-Stage Dynamic Programmingcited by 1
  31. 2023Shrink-Perturb Improves Architecture Mixing during Population Based Training for Neural Architecture Searchcited by 1
  32. 2023SensLI: Sensitivity-Based Layer Insertion for Neural Networkscited by 1
  33. 2024FedTrans: Efficient Federated Learning via Multi-Model Transformationcited by 1
  34. 2024Towards a More Complete Theory of Function Preserving Transformscited by 1
  35. 2025Growing with Experience: Growing Neural Networks in Deep Reinforcement Learningcited by 1
  36. 2026Grow, Don't Overwrite: Fine-tuning Without Forgettingcited by 1
  37. 2026Preservation Is Not Enough for Width Growth: Regime-Sensitive Selection of Dense LM Warm Startscited by 1
  38. 2021Growing Representation Learningcited by 0

50 most cited of 63 All 63 (JSON)