# QuickNets: Saving Training and Preventing Overconfidence in Early-Exit Neural Architectures

arXiv: https://arxiv.org/abs/2212.12866 · HTML: https://mlsanity.com/paper/2212.12866/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 41 references that are in the map.
- First posted on arXiv 25 Dec 2022.

## Builds on (most cited first, 26 of 26)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms](https://mlsanity.com/paper/1708.07747/) (2017)
- [Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding](https://mlsanity.com/paper/1510.00149/) (2015)
- [BranchyNet: Fast Inference via Early Exiting from Deep Neural Networks](https://mlsanity.com/paper/1709.01686/) (2017)
- [SkipNet: Learning Dynamic Routing in Convolutional Networks](https://mlsanity.com/paper/1711.09485/) (2017)
- [BERT Loses Patience: Fast and Robust Inference with Early Exit](https://mlsanity.com/paper/2006.04152/) (2020)
- [BlockDrop: Dynamic Inference Paths in Residual Networks](https://mlsanity.com/paper/1711.08393/) (2017)
- [Convolutional Networks with Adaptive Inference Graphs](https://mlsanity.com/paper/1711.11503/) (2017)
- [Training Neural Networks with Local Error Signals](https://mlsanity.com/paper/1901.06656/) (2019)
- [Greedy Layerwise Learning Can Scale to ImageNet](https://mlsanity.com/paper/1812.11446/) (2018)
- [Layerwise learning for quantum neural networks](https://mlsanity.com/paper/2006.14904/) (2020)
- [Decoupled Greedy Learning of CNNs](https://mlsanity.com/paper/1901.08164/) (2019)
- [Why should we add early exits to neural networks?](https://mlsanity.com/paper/2004.12814/) (2020)
- [Conditional Deep Learning for Energy-Efficient and Enhanced Pattern Recognition](https://mlsanity.com/paper/1509.08971/) (2015)
- [Improved Techniques for Training Adaptive Deep Networks](https://mlsanity.com/paper/1908.06294/) (2019)
- [FrameExit: Conditional Early Exiting for Efficient Video Recognition](https://mlsanity.com/paper/2104.13400/) (2021)
- [IDK Cascades: Fast Deep Learning by Learning not to Overthink](https://mlsanity.com/paper/1706.00885/) (2017)
- [Deep supervised learning using local errors](https://mlsanity.com/paper/1711.06756/) (2017)
- [Generalizing Pooling Functions in Convolutional Neural Networks: Mixed, Gated, and Tree](https://mlsanity.com/paper/1509.08985/) (2015)
- [Learning to Cascade: Confidence Calibration for Improving the Accuracy and Computational Cost of Cascade Inference Systems](https://mlsanity.com/paper/2104.09286/) (2021)
- [Improving the Accuracy of Early Exits in Multi-Exit Architectures via Curriculum Learning](https://mlsanity.com/paper/2104.10461/) (2021)
- [Elastic Neural Networks for Classification](https://mlsanity.com/paper/1810.00589/) (2018)
- [Why Layer-Wise Learning is Hard to Scale-up and a Possible Solution via Accelerated Downsampling](https://mlsanity.com/paper/2010.08038/) (2020)
- [Train your classifier first: Cascade Neural Networks Training from upper layers to lower layers](https://mlsanity.com/paper/2102.04697/) (2021)

## Built on by (most cited first, 1 of 1)

- [Optimizing Attention and Cognitive Control Costs Using Temporally-Layered Architectures](https://mlsanity.com/paper/2305.18701/) (2023)

## Abstract (arXiv)

Deep neural networks have long training and processing times. Early exits added to neural networks allow the network to make early predictions using intermediate activations in the network in time-sensitive applications. However, early exits increase the training time of the neural networks. We introduce QuickNets: a novel cascaded training algorithm for faster training of neural networks. QuickNets are trained in a layer-wise manner such that each successive layer is only trained on samples that could not be correctly classified by the previous layers. We demonstrate that QuickNets can dynamically distribute learning and have a reduced training cost and inference cost compared to standard Backpropagation. Additionally, we introduce commitment layers that significantly improve the early exits by identifying for over-confident predictions and demonstrate its success.