# SensLI: Sensitivity-Based Layer Insertion for Neural Networks

arXiv: https://arxiv.org/abs/2311.15995 · HTML: https://mlsanity.com/paper/2311.15995/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 33 references that are in the map.
- First posted on arXiv 27 Nov 2023.

## Builds on (most cited first, 18 of 18)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Optimization Methods for Large-Scale Machine Learning](https://mlsanity.com/paper/1606.04838/) (2016)
- [Stable Architectures for Deep Neural Networks](https://mlsanity.com/paper/1705.03341/) (2017)
- [Net2Net: Accelerating Learning via Knowledge Transfer](https://mlsanity.com/paper/1511.05641/) (2015)
- [The Singular Values of Convolutional Layers](https://mlsanity.com/paper/1805.10408/) (2018)
- [MorphNet: Fast & Simple Resource-Constrained Structure Learning of Deep Networks](https://mlsanity.com/paper/1711.06798/) (2017)
- [GradMax: Growing Neural Networks using Gradient Information](https://mlsanity.com/paper/2201.05125/) (2022)
- [Network Morphism](https://mlsanity.com/paper/1603.01670/) (2016)
- [NeST: A Neural Network Synthesis Tool Based on a Grow-and-Prune Paradigm](https://mlsanity.com/paper/1711.02017/) (2017)
- [Multi-level Residual Networks from Dynamical Systems View](https://mlsanity.com/paper/1710.10348/) (2017)
- [Splitting Steepest Descent for Growing Neural Architectures](https://mlsanity.com/paper/1910.02366/) (2019)
- [AdaNet: Adaptive Structural Learning of Artificial Neural Networks](https://mlsanity.com/paper/1607.01097/) (2016)
- [Firefly Neural Architecture Descent: a General Approach for Growing Neural Networks](https://mlsanity.com/paper/2102.08574/) (2021)
- [AutoGrow: Automatic Layer Growing in Deep Convolutional Networks](https://mlsanity.com/paper/1906.02909/) (2019)
- [When, where, and how to add new neurons to ANNs](https://mlsanity.com/paper/2202.08539/) (2022)
- [Self-Expanding Neural Networks](https://mlsanity.com/paper/2307.04526/) (2023)

## Built on by (most cited first, 1 of 1)

- [An optimal control approach for neural network architecture adaptation with a posteriori error estimation](https://mlsanity.com/paper/2607.07637/) (2026)

## Abstract (arXiv)

The training of neural networks requires tedious and often manual tuning of the network architecture. We propose a systematic approach to inserting new layers during the training process. Our method eliminates the need to choose a fixed network size before training, is numerically inexpensive to execute and applicable to various architectures including fully connected feedforward networks, ResNets and CNNs. Our technique borrows ideas from constrained optimization and is based on first-order sensitivity information of the loss function with respect to the virtual parameters that additional layers, if inserted, would offer. In numerical experiments, our proposed sensitivity-based layer insertion technique (SensLI) exhibits improved performance on training loss and test error, compared to training on a fixed architecture, and reduced computational effort in comparison to training the extended architecture from the beginning. Our code is available on https://github.com/mathemml/SensLI.