# Learning to Weight Samples for Dynamic Early-exiting Networks

arXiv: https://arxiv.org/abs/2209.08310 · HTML: https://mlsanity.com/paper/2209.08310/

## Key facts

- Cited by 44 papers in the map (data through 7 Oct 2026).
- Placed in Early exit and edge inference, one of 12 directions in Efficient LLM inference and serving.
- Builds on 31 of its 52 references that are in the map.
- First posted on arXiv 17 Sep 2022.

## Placed in

- Efficient LLM inference and serving / [Early exit and edge inference](https://mlsanity.com/inference/early-exit-and-edge-inference/)

## Builds on (most cited first, 30 of 31)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://mlsanity.com/paper/1703.03400/) (2017)
- [Going Deeper with Convolutions](https://mlsanity.com/paper/1409.4842/) (2014)
- [MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications](https://mlsanity.com/paper/1704.04861/) (2017)
- [Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding](https://mlsanity.com/paper/1510.00149/) (2015)
- [Class-Balanced Loss Based on Effective Number of Samples](https://mlsanity.com/paper/1901.05555/) (2019)
- [ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devices](https://mlsanity.com/paper/1707.01083/) (2017)
- [Pruning Filters for Efficient ConvNets](https://mlsanity.com/paper/1608.08710/) (2016)
- [Meta-Learning in Neural Networks: A Survey](https://mlsanity.com/paper/2004.05439/) (2020)
- [Learning Efficient Convolutional Networks through Network Slimming](https://mlsanity.com/paper/1708.06519/) (2017)
- [Learning to Reweight Examples for Robust Deep Learning](https://mlsanity.com/paper/1803.09050/) (2018)
- [BranchyNet: Fast Inference via Early Exiting from Deep Neural Networks](https://mlsanity.com/paper/1709.01686/) (2017)
- [Dynamic Neural Networks: A Survey](https://mlsanity.com/paper/2102.04906/) (2021)
- [Filter Pruning via Geometric Median for Deep Convolutional Neural Networks Acceleration](https://mlsanity.com/paper/1811.00250/) (2018)
- [SkipNet: Learning Dynamic Routing in Convolutional Networks](https://mlsanity.com/paper/1711.09485/) (2017)
- [Data-Driven Sparse Structure Selection for Deep Neural Networks](https://mlsanity.com/paper/1707.01213/) (2017)
- [Adaptive Neural Networks for Efficient Inference](https://mlsanity.com/paper/1702.07811/) (2017)
- [Resolution Adaptive Networks for Efficient Inference](https://mlsanity.com/paper/2003.07326/) (2020)
- [Dynamic Convolutions: Exploiting Spatial Sparsity for Faster Inference](https://mlsanity.com/paper/1912.03203/) (2019)
- [Network Pruning via Transformable Architecture Search](https://mlsanity.com/paper/1905.09717/) (2019)
- [Learning Fast Sample Re-weighting Without Reward Data](https://mlsanity.com/paper/2109.03216/) (2021)
- [Conditional Deep Learning for Energy-Efficient and Enhanced Pattern Recognition](https://mlsanity.com/paper/1509.08971/) (2015)
- [Improved Techniques for Training Adaptive Deep Networks](https://mlsanity.com/paper/1908.06294/) (2019)
- [Batch-Shaping for Learning Conditional Channel Gated Networks](https://mlsanity.com/paper/1907.06627/) (2019)
- [CondenseNet V2: Sparse Feature Reactivation for Deep Networks](https://mlsanity.com/paper/2104.04382/) (2021)

## Built on by (most cited first, 30 of 44)

- [FLatten Transformer: Vision Transformer using Focused Linear Attention](https://mlsanity.com/paper/2308.00442/) (2023)
- [GSVA: Generalized Segmentation via Multimodal Large Language Models](https://mlsanity.com/paper/2312.10103/) (2023)
- [Weakly-Supervised Concealed Object Segmentation with SAM-based Pseudo Labeling and Multi-scale Feature Grouping](https://mlsanity.com/paper/2305.11003/) (2023)
- [Agent Attention: On the Integration of Softmax and Linear Attention](https://mlsanity.com/paper/2312.08874/) (2023)
- [Dynamic Diffusion Transformer](https://mlsanity.com/paper/2410.03456/) (2024)
- [Adaptive Rotated Convolution for Rotated Object Detection](https://mlsanity.com/paper/2303.07820/) (2023)
- [Rank-DETR for High Quality Object Detection](https://mlsanity.com/paper/2310.08854/) (2023)
- [EfficientTrain: Exploring Generalized Curriculum Learning for Training Visual Backbones](https://mlsanity.com/paper/2211.09703/) (2022)
- [ART: Anonymous Region Transformer for Variable Multi-Layer Transparent Image Generation](https://mlsanity.com/paper/2502.18364/) (2025)
- [A Stitch in Time Saves Nine: Small VLM is a Precise Guidance for Accelerating Large VLMs](https://mlsanity.com/paper/2412.03324/) (2024)
- [Dynamic Perceiver for Efficient Visual Recognition](https://mlsanity.com/paper/2306.11248/) (2023)
- [Fast yet Safe: Early-Exiting with Risk Control](https://mlsanity.com/paper/2405.20915/) (2024)
- [Latency-aware Unified Dynamic Networks for Efficient Image Recognition](https://mlsanity.com/paper/2308.15949/) (2023)
- [Train Once, Get a Family: State-Adaptive Balances for Offline-to-Online Reinforcement Learning](https://mlsanity.com/paper/2310.17966/) (2023)
- [Real-world Image Dehazing with Coherence-based Pseudo Labeling and Cooperative Unfolding Network](https://mlsanity.com/paper/2406.07966/) (2024)
- [Latency-aware Spatial-wise Dynamic Networks](https://mlsanity.com/paper/2210.06223/) (2022)
- [Dynamic Tuning Towards Parameter and Inference Efficiency for ViT Adaptation](https://mlsanity.com/paper/2403.11808/) (2024)
- [Efficient Diffusion Transformer with Step-wise Dynamic Attention Mediators](https://mlsanity.com/paper/2408.05710/) (2024)
- [Understanding, Predicting and Better Resolving Q-Value Divergence in Offline-RL](https://mlsanity.com/paper/2310.04411/) (2023)
- [DyFADet: Dynamic Feature Aggregation for Temporal Action Detection](https://mlsanity.com/paper/2407.03197/) (2024)
- [Fine-grained Recognition with Learnable Semantic Data Augmentation](https://mlsanity.com/paper/2309.00399/) (2023)
- [Towards Anytime Classification in Early-Exit Architectures by Enforcing Conditional Monotonicity](https://mlsanity.com/paper/2306.02652/) (2023)
- [ROPO: Robust Preference Optimization for Large Language Models](https://mlsanity.com/paper/2404.04102/) (2024)
- [Jointly-Learned Exit and Inference for a Dynamic Neural Network : JEI-DNN](https://mlsanity.com/paper/2310.09163/) (2023)
- [Computation-efficient Deep Learning for Computer Vision: A Survey](https://mlsanity.com/paper/2308.13998/) (2023)
- [Deep Incubation: Training Large Models by Divide-and-Conquering](https://mlsanity.com/paper/2212.04129/) (2022)
- [GRA: Detecting Oriented Objects through Group-wise Rotating and Attention](https://mlsanity.com/paper/2403.11127/) (2024)
- [Conditional computation in neural networks: principles and research trends](https://mlsanity.com/paper/2403.07965/) (2024)
- [Early-Exit with Class Exclusion for Efficient Inference of Neural Networks](https://mlsanity.com/paper/2309.13443/) (2023)
- [ENAT: Rethinking Spatial-temporal Interactions in Token-based Image Synthesis](https://mlsanity.com/paper/2411.06959/) (2024)

## Abstract (arXiv)

Early exiting is an effective paradigm for improving the inference efficiency of deep networks. By constructing classifiers with varying resource demands (the exits), such networks allow easy samples to be output at early exits, removing the need for executing deeper layers. While existing works mainly focus on the architectural design of multi-exit networks, the training strategies for such models are largely left unexplored. The current state-of-the-art models treat all samples the same during training. However, the early-exiting behavior during testing has been ignored, leading to a gap between training and testing. In this paper, we propose to bridge this gap by sample weighting. Intuitively, easy samples, which generally exit early in the network during inference, should contribute more to training early classifiers. The training of hard samples (mostly exit from deeper layers), however, should be emphasized by the late classifiers. Our work proposes to adopt a weight prediction network to weight the loss of different training samples at each exit. This weight prediction network and the backbone model are jointly optimized under a meta-learning framework with a novel optimization objective. By bringing the adaptive behavior during inference into the training phase, we show that the proposed weighting mechanism consistently improves the trade-off between classification accuracy and inference efficiency. Code is available at https://github.com/LeapLabTHU/L2W-DEN.