# Noisy Heuristics NAS: A Network Morphism based Neural Architecture Search using Heuristics

arXiv: https://arxiv.org/abs/2207.04467 · HTML: https://mlsanity.com/paper/2207.04467/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 28 references that are in the map.
- First posted on arXiv 10 Jul 2022.

## Builds on (most cited first, 20 of 20)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [DARTS: Differentiable Architecture Search](https://mlsanity.com/paper/1806.09055/) (2018)
- [Neural Architecture Search with Reinforcement Learning](https://mlsanity.com/paper/1611.01578/) (2016)
- [Energy and Policy Considerations for Deep Learning in NLP](https://mlsanity.com/paper/1906.02243/) (2019)
- [A Comprehensive Survey on Transfer Learning](https://mlsanity.com/paper/1911.02685/) (2019)
- [Importance Estimation for Neural Network Pruning](https://mlsanity.com/paper/1906.10771/) (2019)
- [Designing Neural Network Architectures using Reinforcement Learning](https://mlsanity.com/paper/1611.02167/) (2016)
- [PC-DARTS: Partial Channel Connections for Memory-Efficient Architecture Search](https://mlsanity.com/paper/1907.05737/) (2019)
- [Deep Learning in Spiking Neural Networks](https://mlsanity.com/paper/1804.08150/) (2018)
- [NISP: Pruning Networks using Neuron Importance Score Propagation](https://mlsanity.com/paper/1711.05908/) (2017)
- [Auto-Keras: An Efficient Neural Architecture Search System](https://mlsanity.com/paper/1806.10282/) (2018)
- [MorphNet: Fast & Simple Resource-Constrained Structure Learning of Deep Networks](https://mlsanity.com/paper/1711.06798/) (2017)
- [GradMax: Growing Neural Networks using Gradient Information](https://mlsanity.com/paper/2201.05125/) (2022)
- [NeST: A Neural Network Synthesis Tool Based on a Grow-and-Prune Paradigm](https://mlsanity.com/paper/1711.02017/) (2017)
- [Simple And Efficient Architecture Search for Convolutional Neural Networks](https://mlsanity.com/paper/1711.04528/) (2017)
- [Incremental Learning Using a Grow-and-Prune Paradigm with Efficient Neural Networks](https://mlsanity.com/paper/1905.10952/) (2019)
- [Large Scale Evolution of Convolutional Neural Networks Using Volunteer Computing](https://mlsanity.com/paper/1703.05422/) (2017)
- [CompNet: Neural networks growing via the compact network morphism](https://mlsanity.com/paper/1804.10316/) (2018)

## Built on by (most cited first, 2 of 2)

- [Importance Estimation with Random Gradient for Neural Network Pruning](https://mlsanity.com/paper/2310.20203/) (2023)
- [Metric as Transform: Exploring beyond Affine Transform for Interpretable Neural Network](https://mlsanity.com/paper/2410.16159/) (2024)

## Abstract (arXiv)

Network Morphism based Neural Architecture Search (NAS) is one of the most efficient methods, however, knowing where and when to add new neurons or remove dis-functional ones is generally left to black-box Reinforcement Learning models. In this paper, we present a new Network Morphism based NAS called Noisy Heuristics NAS which uses heuristics learned from manually developing neural network models and inspired by biological neuronal dynamics. Firstly, we add new neurons randomly and prune away some to select only the best fitting neurons. Secondly, we control the number of layers in the network using the relationship of hidden units to the number of input-output connections. Our method can increase or decrease the capacity or non-linearity of models online which is specified with a few meta-parameters by the user. Our method generalizes both on toy datasets and on real-world data sets such as MNIST, CIFAR-10, and CIFAR-100. The performance is comparable to the hand-engineered architecture ResNet-18 with the similar parameters.