# Efficient Architecture Search by Network Transformation

arXiv: https://arxiv.org/abs/1707.04873 · HTML: https://mlsanity.com/paper/1707.04873/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 Jul 2017.

## Built on by (most cited first, 9 of 9)

- [GradMax: Growing Neural Networks using Gradient Information](https://mlsanity.com/paper/2201.05125/) (2022)
- [Searching for Fast Model Families on Datacenter Accelerators](https://mlsanity.com/paper/2102.05610/) (2021)
- [A Survey on Computationally Efficient Neural Architecture Search](https://mlsanity.com/paper/2206.01520/) (2022)
- [Automatic Attention Pruning: Improving and Automating Model Pruning using Attentions](https://mlsanity.com/paper/2303.08595/) (2023)
- [TeLU Activation Function for Fast and Stable Deep Learning](https://mlsanity.com/paper/2412.20269/) (2024)
- [Adaptive Activation-based Structured Pruning](https://mlsanity.com/paper/2201.10520/) (2022)
- [EASNet: Searching Elastic and Accurate Network Architecture for Stereo Matching](https://mlsanity.com/paper/2207.09796/) (2022)
- [Differentiable Network Adaption with Elastic Search Space](https://mlsanity.com/paper/2103.16350/) (2021)
- [Towards Device Efficient Conditional Image Generation](https://mlsanity.com/paper/2203.10363/) (2022)

## Abstract (arXiv)

Techniques for automatically designing deep neural network architectures such as reinforcement learning based approaches have recently shown promising results. However, their success is based on vast computational resources (e.g. hundreds of GPUs), making them difficult to be widely used. A noticeable limitation is that they still design and train each network from scratch during the exploration of the architecture space, which is highly inefficient. In this paper, we propose a new framework toward efficient architecture search by exploring the architecture space based on the current network and reusing its weights. We employ a reinforcement learning agent as the meta-controller, whose action is to grow the network depth or layer width with function-preserving transformations. As such, the previously validated networks can be reused for further exploration, thus saves a large amount of computational cost. We apply our method to explore the architecture space of the plain convolutional neural networks (no skip-connections, branching etc.) on image benchmark datasets (CIFAR-10, SVHN) with restricted computational resources (5 GPUs). Our method can design highly competitive networks that outperform existing networks using the same design scheme. On CIFAR-10, our model without skip-connections achieves 4.23% test error rate, exceeding a vast majority of modern architectures and approaching DenseNet. Furthermore, by applying our method to explore the DenseNet architecture space, we are able to achieve more accurate networks with fewer parameters.