# Efficient Forward Architecture Search

arXiv: https://arxiv.org/abs/1905.13360 · HTML: https://mlsanity.com/paper/1905.13360/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 31 May 2019.

## Built on by (most cited first, 14 of 14)

- [Neural Architecture Search: Insights from 1000 Papers](https://mlsanity.com/paper/2301.08727/) (2023)
- [Firefly Neural Architecture Descent: a General Approach for Growing Neural Networks](https://mlsanity.com/paper/2102.08574/) (2021)
- [An Evaluation of Edge TPU Accelerators for Convolutional Neural Networks](https://mlsanity.com/paper/2102.10423/) (2021)
- [NAS-Bench-x11 and the Power of Learning Curves](https://mlsanity.com/paper/2111.03602/) (2021)
- [LiteTransformerSearch: Training-free Neural Architecture Search for Efficient Language Models](https://mlsanity.com/paper/2203.02094/) (2022)
- [Reconsidering CO2 emissions from Computer Vision](https://mlsanity.com/paper/2104.08702/) (2021)
- [Fisher Task Distance and Its Application in Neural Architecture Search](https://mlsanity.com/paper/2103.12827/) (2021)
- [Towards Less Constrained Macro-Neural Architecture Search](https://mlsanity.com/paper/2203.05508/) (2022)
- [Exploring the Intersection between Neural Architecture Search and Continual Learning](https://mlsanity.com/paper/2206.05625/) (2022)
- [Frozen Layers: Memory-efficient Many-fidelity Hyperparameter Optimization](https://mlsanity.com/paper/2504.10735/) (2025)
- [Balanced Mixture of SuperNets for Learning the CNN Pooling Architecture](https://mlsanity.com/paper/2306.11982/) (2023)
- [Neural Architecture Search for Inversion](https://mlsanity.com/paper/2201.01772/) (2022)
- [Growing Neural Network with Shared Parameter](https://mlsanity.com/paper/2201.06500/) (2022)
- [Efficient Global Neural Architecture Search](https://mlsanity.com/paper/2502.03553/) (2025)

## Abstract (arXiv)

We propose a neural architecture search (NAS) algorithm, Petridish, to iteratively add shortcut connections to existing network layers. The added shortcut connections effectively perform gradient boosting on the augmented layers. The proposed algorithm is motivated by the feature selection algorithm forward stage-wise linear regression, since we consider NAS as a generalization of feature selection for regression, where NAS selects shortcuts among layers instead of selecting features. In order to reduce the number of trials of possible connection combinations, we train jointly all possible connections at each stage of growth while leveraging feature selection techniques to choose a subset of them. We experimentally show this process to be an efficient forward architecture search algorithm that can find competitive models using few GPU days in both the search space of repeatable network modules (cell-search) and the space of general networks (macro-search). Petridish is particularly well-suited for warm-starting from existing models crucial for lifelong-learning scenarios.