mlsanityData to 7 Oct 2026

QuickNets: Saving Training and Preventing Overconfidence in Early-Exit Neural Architectures

arXiv 2212.12866 ↗First posted 25 Dec 2022cs.LG cs.AI cs.NEcited by 1

Not on the map. Why?

Abstract (arXiv)

Deep neural networks have long training and processing times. Early exits added to neural networks allow the network to make early predictions using intermediate activations in the network in time-sensitive applications. However, early exits increase the training time of the neural networks. We introduce QuickNets: a novel cascaded training algorithm for faster training of neural networks. QuickNets are trained in a layer-wise manner such that each successive layer is only trained on samples that could not be correctly classified by the previous layers. We demonstrate that QuickNets can dynamically distribute learning and have a reduced training cost and inference cost compared to standard Backpropagation. Additionally, we introduce commitment layers that significantly improve the early exits by identifying for over-confident predictions and demonstrate its success.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 26 of its 41 references that are in the map. First posted on arXiv 25 Dec 2022.

Builds on

26 of 41 references.

References · 26

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  3. 2016Densely Connected Convolutional Networkscited by 6,691
  4. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,283
  5. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,291
  6. 2017BranchyNet: Fast Inference via Early Exiting from Deep Neural Networkscited by 591
  7. 2017SkipNet: Learning Dynamic Routing in Convolutional Networkscited by 328
  8. 2020BERT Loses Patience: Fast and Robust Inference with Early Exitcited by 267
  9. 2017BlockDrop: Dynamic Inference Paths in Residual Networkscited by 185
  10. 2017Convolutional Networks with Adaptive Inference Graphscited by 147
  11. 2019Training Neural Networks with Local Error Signalscited by 135
  12. 2018Greedy Layerwise Learning Can Scale to ImageNetcited by 113
14 more
  1. 2020Layerwise learning for quantum neural networkscited by 87
  2. 2019Decoupled Greedy Learning of CNNscited by 76
  3. 2020Why should we add early exits to neural networks?cited by 70
  4. 2015Conditional Deep Learning for Energy-Efficient and Enhanced Pattern Recognitioncited by 67
  5. 2019Improved Techniques for Training Adaptive Deep Networkscited by 66
  6. 2021FrameExit: Conditional Early Exiting for Efficient Video Recognitioncited by 59
  7. 2017IDK Cascades: Fast Deep Learning by Learning not to Overthinkcited by 47
  8. 2017Deep supervised learning using local errorscited by 41
  9. 2015Generalizing Pooling Functions in Convolutional Neural Networks: Mixed, Gated, and Treecited by 36
  10. 2021Learning to Cascade: Confidence Calibration for Improving the Accuracy and Computational Cost of Cascade Inference Systemscited by 14
  11. 2021Improving the Accuracy of Early Exits in Multi-Exit Architectures via Curriculum Learningcited by 9
  12. 2018Elastic Neural Networks for Classificationcited by 5
  13. 2020Why Layer-Wise Learning is Hard to Scale-up and a Possible Solution via Accelerated Downsamplingcited by 3
  14. 2021Train your classifier first: Cascade Neural Networks Training from upper layers to lower layerscited by 1

Built on by

Cited by year

Citing papers per year: 2023: 1.

1 paper.

  1. 2023Optimizing Attention and Cognitive Control Costs Using Temporally-Layered Architecturescited by 2

Similar, not cited

Close in wording, no citation either way

  1. 2022Pruning Early Exit Networks
  2. 2023Towards Anytime Classification in Early-Exit Architectures by Enforcing Conditional Monotonicity
  3. 2024Fast yet Safe: Early-Exiting with Risk Control
  4. 2022Learning to Weight Samples for Dynamic Early-exiting Networks
  5. 2021Adaptive Inference through Early-Exit Networks: Design, Challenges and Directions
  6. 2023Adaptive Deep Neural Network Inference Optimization with EENet
  7. 2022Boosted Dynamic Neural Networks
  8. 2022Binary Early-Exit Network for Adaptive Inference on Low-Resource Devices