# Hierarchical Training of Deep Neural Networks Using Early Exiting

arXiv: https://arxiv.org/abs/2303.02384 · HTML: https://mlsanity.com/paper/2303.02384/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 37 references that are in the map.
- First posted on arXiv 4 Mar 2023.

## Builds on (most cited first, 21 of 21)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [You Only Look Once: Unified, Real-Time Object Detection](https://mlsanity.com/paper/1506.02640/) (2015)
- [Going Deeper with Convolutions](https://mlsanity.com/paper/1409.4842/) (2014)
- [Rich feature hierarchies for accurate object detection and semantic segmentation](https://mlsanity.com/paper/1311.2524/) (2013)
- [Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation](https://mlsanity.com/paper/1308.3432/) (2013)
- [OpenPose: Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields](https://mlsanity.com/paper/1812.08008/) (2018)
- [DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients](https://mlsanity.com/paper/1606.06160/) (2016)
- [A White Paper on Neural Network Quantization](https://mlsanity.com/paper/2106.08295/) (2021)
- [Distributed Deep Neural Networks over the Cloud, the Edge and End Devices](https://mlsanity.com/paper/1709.01921/) (2017)
- [JointDNN: An Efficient Training and Inference Engine for Intelligent Mobile Cloud Computing Services](https://mlsanity.com/paper/1801.08618/) (2018)
- [Privacy in Deep Learning: A Survey](https://mlsanity.com/paper/2004.12254/) (2020)
- [Flexpoint: An Adaptive Numerical Format for Efficient Training of Deep Neural Networks](https://mlsanity.com/paper/1711.02213/) (2017)
- [JALAD: Joint Accuracy- and Latency-Aware Deep Structure Decoupling for Edge-Cloud Execution](https://mlsanity.com/paper/1812.10027/) (2018)
- [Structure preserving deep learning](https://mlsanity.com/paper/2006.03364/) (2020)
- [Shifted and Squeezed 8-bit Floating Point format for Low-Precision Training of Deep Neural Networks](https://mlsanity.com/paper/2001.05674/) (2020)
- [Fathom: Reference Workloads for Modern Deep Learning Methods](https://mlsanity.com/paper/1608.06581/) (2016)
- [HierTrain: Fast Hierarchical Edge AI Learning with Hybrid Parallelism in Mobile-Edge-Cloud Computing](https://mlsanity.com/paper/2003.09876/) (2020)
- [p-Meta: Towards On-device Deep Model Adaptation](https://mlsanity.com/paper/2206.12705/) (2022)

## Built on by (most cited first, 5 of 5)

- [Image Coding for Machines via Feature-Preserving Rate-Distortion Optimization](https://mlsanity.com/paper/2504.02216/) (2025)
- [PriPHiT: Privacy-Preserving Hierarchical Training of Deep Neural Networks](https://mlsanity.com/paper/2408.05092/) (2024)
- [EPSILON: Adaptive Fault Mitigation in Approximate Deep Neural Network using Statistical Signatures](https://mlsanity.com/paper/2504.20074/) (2025)
- [Hetero-SplitEE: Split Learning of Neural Networks with Early Exits for Heterogeneous IoT Devices](https://mlsanity.com/paper/2512.09313/) (2025)
- [ECA-BLS: An Efficient Complex-Augmented Broad Learning System](https://mlsanity.com/paper/2608.29763/) (2026)

## Abstract (arXiv)

Deep neural networks provide state-of-the-art accuracy for vision tasks but they require significant resources for training. Thus, they are trained on cloud servers far from the edge devices that acquire the data. This issue increases communication cost, runtime and privacy concerns. In this study, a novel hierarchical training method for deep neural networks is proposed that uses early exits in a divided architecture between edge and cloud workers to reduce the communication cost, training runtime and privacy concerns. The method proposes a brand-new use case for early exits to separate the backward pass of neural networks between the edge and the cloud during the training phase. We address the issues of most available methods that due to the sequential nature of the training phase, cannot train the levels of hierarchy simultaneously or they do it with the cost of compromising privacy. In contrast, our method can use both edge and cloud workers simultaneously, does not share the raw input data with the cloud and does not require communication during the backward pass. Several simulations and on-device experiments for different neural network architectures demonstrate the effectiveness of this method. It is shown that the proposed method reduces the training runtime for VGG-16 and ResNet-18 architectures by 29% and 61% in CIFAR-10 classification and by 25% and 81% in Tiny ImageNet classification when the communication with the cloud is done over a low bit rate channel. This gain in the runtime is achieved whilst the accuracy drop is negligible. This method is advantageous for online learning of high-accuracy deep neural networks on sensor-holding low-resource devices such as mobile phones or robots as a part of an edge-cloud system, making them more flexible in facing new tasks and classes of data.