# How to Train Your Metamorphic Deep Neural Network

arXiv: https://arxiv.org/abs/2505.05510 · HTML: https://mlsanity.com/paper/2505.05510/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 27 of its 36 references that are in the map.
- First posted on arXiv 7 May 2025.

## Builds on (most cited first, 27 of 27)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks](https://mlsanity.com/paper/1803.03635/) (2018)
- [Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains](https://mlsanity.com/paper/2006.10739/) (2020)
- [FitNets: Hints for Thin Deep Nets](https://mlsanity.com/paper/1412.6550/) (2014)
- [On the Spectral Bias of Neural Networks](https://mlsanity.com/paper/1806.08734/) (2018)
- [Pruning Filters for Efficient ConvNets](https://mlsanity.com/paper/1608.08710/) (2016)
- [Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)](https://mlsanity.com/paper/1511.07289/) (2015)
- [Dark Experience for General Continual Learning: a Strong, Simple Baseline](https://mlsanity.com/paper/2004.07211/) (2020)
- [Once-for-All: Train One Network and Specialize it for Efficient Deployment](https://mlsanity.com/paper/1908.09791/) (2019)
- [Block-NeRF: Scalable Large Scene Neural View Synthesis](https://mlsanity.com/paper/2202.05263/) (2022)
- [KiloNeRF: Speeding up Neural Radiance Fields with Thousands of Tiny MLPs](https://mlsanity.com/paper/2103.13744/) (2021)
- [Deep Mutual Learning](https://mlsanity.com/paper/1706.00384/) (2017)
- [Learning Structured Sparsity in Deep Neural Networks](https://mlsanity.com/paper/1608.03665/) (2016)
- [RepVGG: Making VGG-style ConvNets Great Again](https://mlsanity.com/paper/2101.03697/) (2021)
- [Be Your Own Teacher: Improve the Performance of Convolutional Neural Networks via Self Distillation](https://mlsanity.com/paper/1905.08094/) (2019)
- [Soft Filter Pruning for Accelerating Deep Convolutional Neural Networks](https://mlsanity.com/paper/1808.06866/) (2018)
- [Slimmable Neural Networks](https://mlsanity.com/paper/1812.08928/) (2018)
- [DynaBERT: Dynamic BERT with Adaptive Width and Depth](https://mlsanity.com/paper/2004.04037/) (2020)
- [Universally Slimmable Networks and Improved Training Techniques](https://mlsanity.com/paper/1903.05134/) (2019)
- [DepGraph: Towards Any Structural Pruning](https://mlsanity.com/paper/2301.12900/) (2023)
- [Understanding Batch Normalization](https://mlsanity.com/paper/1806.02375/) (2018)
- [Vision Transformer Slimming: Multi-Dimension Searching in Continuous Optimization Space](https://mlsanity.com/paper/2201.00814/) (2022)
- [A Machine Learning-oriented Survey on Tiny Machine Learning](https://mlsanity.com/paper/2309.11932/) (2023)
- [Dynamic ConvNets on Tiny Devices via Nested Sparsity](https://mlsanity.com/paper/2203.03324/) (2022)
- [Neural Metamorphosis](https://mlsanity.com/paper/2410.11878/) (2024)

## Abstract (arXiv)

Neural Metamorphosis (NeuMeta) is a recent paradigm for generating neural networks of varying width and depth. Based on Implicit Neural Representation (INR), NeuMeta learns a continuous weight manifold, enabling the direct generation of compressed models, including those with configurations not seen during training. While promising, the original formulation of NeuMeta proves effective only for the final layers of the undelying model, limiting its broader applicability. In this work, we propose a training algorithm that extends the capabilities of NeuMeta to enable full-network metamorphosis with minimal accuracy degradation. Our approach follows a structured recipe comprising block-wise incremental training, INR initialization, and strategies for replacing batch normalization. The resulting metamorphic networks maintain competitive accuracy across a wide range of compression ratios, offering a scalable solution for adaptable and efficient deployment of deep models. The code is available at: https://github.com/TSommariva/HTTY_NeuMeta.