# Adaptive Neural Networks Using Residual Fitting

arXiv: https://arxiv.org/abs/2301.05744 · HTML: https://mlsanity.com/paper/2301.05744/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 8 of its 15 references that are in the map.
- First posted on arXiv 13 Jan 2023.

## Builds on (most cited first, 8 of 8)

- [The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks](https://mlsanity.com/paper/1803.03635/) (2018)
- [Generative Adversarial Imitation Learning](https://mlsanity.com/paper/1606.03476/) (2016)
- [What is the State of Neural Network Pruning?](https://mlsanity.com/paper/2003.03033/) (2020)
- [The Evolved Transformer](https://mlsanity.com/paper/1901.11117/) (2019)
- [GradMax: Growing Neural Networks using Gradient Information](https://mlsanity.com/paper/2201.05125/) (2022)
- [AdaNet: Adaptive Structural Learning of Artificial Neural Networks](https://mlsanity.com/paper/1607.01097/) (2016)
- [Towards Searching Efficient and Accurate Neural Network Architectures in Binary Classification Problems](https://mlsanity.com/paper/2101.06511/) (2021)
- [Self-adaptive deep neural network: Numerical approximation to functions and PDEs](https://mlsanity.com/paper/2109.02839/) (2021)

## Abstract (arXiv)

Current methods for estimating the required neural-network size for a given problem class have focused on methods that can be computationally intensive, such as neural-architecture search and pruning. In contrast, methods that add capacity to neural networks as needed may provide similar results to architecture search and pruning, but do not require as much computation to find an appropriate network size. Here, we present a network-growth method that searches for explainable error in the network's residuals and grows the network if sufficient error is detected. We demonstrate this method using examples from classification, imitation learning, and reinforcement learning. Within these tasks, the growing network can often achieve better performance than small networks that do not grow, and similar performance to networks that begin much larger.