Data to 14 Oct 2026

Improved Techniques for Training Adaptive Deep Networks

arXiv 1908.06294 ↗First posted 17 Aug 2019cs.CVcited by 66

Not on the map. Why?

Abstract (arXiv)

Adaptive inference is a promising technique to improve the computational efficiency of deep models at test time. In contrast to static models which use the same computation graph for all instances, adaptive networks can dynamically adjust their structure conditioned on each input. While existing research on adaptive inference mainly focuses on designing more advanced architectures, this paper investigates how to train such networks more effectively. Specifically, we consider a typical adaptive deep network with multiple intermediate classifiers. We present three techniques to improve its training efficacy from two aspects: 1) a Gradient Equilibrium algorithm to resolve the conflict of learning of different classifiers; 2) an Inline Subnetwork Collaboration approach and a One-for-all Knowledge Distillation algorithm to enhance the collaboration among classifiers. On multiple datasets (CIFAR-10, CIFAR-100 and ImageNet), we show that the proposed approach consistently leads to further improved efficiency on top of state-of-the-art adaptive deep networks.

Key facts. Cited by 66 papers in the map (data through 14 Oct 2026). First posted on arXiv 17 Aug 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 17; 2022: 10; 2023: 21; 2024: 13; 2025: 3; 2026: 2.

66 papers.

Sort:

By area:

2021Dynamic Neural Networks: A Surveycited by 408
2021Pruning and Quantization for Deep Neural Network Acceleration: A Surveycited by 263
2021AdaViT: Adaptive Vision Transformers for Efficient Image Recognitioncited by 228
2021Not All Images are Worth 16x16 Words: Dynamic Transformers for Efficient Image Recognitioncited by 144
2021Split Computing and Early Exiting for Deep Learning Applications: Survey and Research Challengescited by 115
2021MOOD: Multi-level Out-of-distribution Detectioncited by 94
2021Dynamic Slimmable Networkcited by 83
2021Spatially-Adaptive Image Restoration using Distortion-Guided Networkscited by 77
2021Adaptive Inference through Early-Exit Networks: Design, Challenges and Directionscited by 75
2021Towards Efficient NLP: A Standard Evaluation and A Strong Baselinecited by 44
2022Learning to Weight Samples for Dynamic Early-exiting Networkscited by 44
2022You Need Multiple Exiting: Dynamic Early Exiting for Accelerating Unified Vision Language Modelcited by 43

38 more

2022Glance and Focus Networks for Dynamic Visual Recognitioncited by 41
2021Zero Time Waste: Recycling Predictions in Early Exit Neural Networkscited by 39
2022Automated Progressive Learning for Efficient Training of Vision Transformerscited by 37
2023Dynamic Perceiver for Efficient Visual Recognitioncited by 33
2021Single-Layer Vision Transformers for More Accurate Early Exits with Less Overheadcited by 26
2022SplitGP: Achieving Both Generalization and Personalization in Federated Learningcited by 20
2023Fixing Overconfidence in Dynamic Neural Networkscited by 19
2021Multi-Exit Semantic Segmentation Networkscited by 18
2023Towards Anytime Classification in Early-Exit Architectures by Enforcing Conditional Monotonicitycited by 16
2023DynamicDet: A Unified Dynamic Architecture for Object Detectioncited by 15
2023Jointly-Learned Exit and Inference for a Dynamic Neural Network : JEI-DNNcited by 15
2023Computation-efficient Deep Learning for Computer Vision: A Surveycited by 14
2024LF-ViT: Reducing Spatial Redundancy in Vision Transformer for Efficient Image Recognitioncited by 13
2024Recurrent Early Exits for Federated Learning with Heterogeneous Clientscited by 12
2023Adaptive Deep Neural Network Inference Optimization with EENetcited by 9
2022Boosted Dynamic Neural Networkscited by 8
2023Early-Exit with Class Exclusion for Efficient Inference of Neural Networkscited by 8
2023Dynamic nsNet2: Efficient Deep Noise Suppression with Early Exitingcited by 7
2021DS-Net++: Dynamic Weight Slicing for Efficient Inference in CNNs and Transformerscited by 6
2023Early-Exit Neural Networks with Nested Prediction Setscited by 6
2024Efficient Training of Large Vision Models via Advanced Automated Progressive Learningcited by 5
2025A Survey on Dynamic Neural Networks: from Computer Vision to Multi-modal Sensor Fusioncited by 5
2021Students are the Best Teacher: Exit-Ensemble Distillation with Multi-Exitscited by 4
2023VERSE: Virtual-Gradient Aware Streaming Lifelong Learning with Anytime Inferencecited by 4
2022A Closer Look at Branch Classifiers of Multi-exit Architecturescited by 3
2023Learning Task-preferred Inference Routes for Gradient De-conflict in Multi-output DNNscited by 3
2023RAI4IoE: Responsible AI for Enabling the Internet of Energycited by 3
2023NEO-KD: Knowledge-Distillation-Based Adversarial Training for Robust Multi-Exit Neural Networkscited by 3
2024On the Impact of Black-box Deployment Strategies for Edge AI on Latency and Model Performancecited by 3
2021HCMS: Hierarchical and Conditional Modality Selection for Efficient Video Recognitioncited by 2
2021Complexity-aware Adaptive Training and Inference for Edge-Cloud Distributed AI Systemscited by 2
2022COST-EFF: Collaborative Optimization of Spatial and Temporal Efficiency with Slenderized Multi-exit Language Modelscited by 2
2024Multimodal Adaptive Inference for Document Image Classification with Anytime Early Exitingcited by 2
2024Deep Feature Surgery: Towards Accurate and Efficient Multi-Exit Networkscited by 2
2024Multiple-Exit Tuning: Towards Inference-Efficient Adaptation for Vision Transformercited by 2
2022QuickNets: Saving Training and Preventing Overconfidence in Early-Exit Neural Architecturescited by 1
2023Learning to Compose SuperWeights for Neural Parameter Allocation Searchcited by 1
2023Progressive Painterly Image Harmonization from Low-level Styles to High-level Stylescited by 1

50 most cited of 66 All 66 (JSON)