# Dynamic Deep Neural Networks: Optimizing Accuracy-Efficiency Trade-offs by Selective Execution

arXiv: https://arxiv.org/abs/1701.00299 · HTML: https://mlsanity.com/paper/1701.00299/

## Key facts

- Cited by 59 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Jan 2017.

## Built on by (most cited first, 30 of 59)

- [Dynamic Neural Networks: A Survey](https://mlsanity.com/paper/2102.04906/) (2021)
- [AdaViT: Adaptive Tokens for Efficient Vision Transformer](https://mlsanity.com/paper/2112.07658/) (2021)
- [Adaptive Inference through Early-Exit Networks: Design, Challenges and Directions](https://mlsanity.com/paper/2106.05022/) (2021)
- [FedCP: Separating Feature Information for Personalized Federated Learning via Conditional Policy](https://mlsanity.com/paper/2307.01217/) (2023)
- [Efficient Split-Mix Federated Learning for On-Demand and In-Situ Customization](https://mlsanity.com/paper/2203.09747/) (2022)
- [FOVEA: Foveated Image Magnification for Autonomous Navigation](https://mlsanity.com/paper/2108.12102/) (2021)
- [Enable Deep Learning on Mobile Devices: Methods, Systems, and Applications](https://mlsanity.com/paper/2204.11786/) (2022)
- [Visual Instance-aware Prompt Tuning](https://mlsanity.com/paper/2507.07796/) (2025)
- [Coarse-Fine Networks for Temporal Activity Detection in Videos](https://mlsanity.com/paper/2103.01302/) (2021)
- [MicroNet: Improving Image Recognition with Extremely Low FLOPs](https://mlsanity.com/paper/2108.05894/) (2021)
- [Fire Together Wire Together: A Dynamic Pruning Approach with Self-Supervised Mask Prediction](https://mlsanity.com/paper/2110.08232/) (2021)
- [Dynamic Spatial Sparsification for Efficient Vision Transformers and Convolutional Neural Networks](https://mlsanity.com/paper/2207.01580/) (2022)
- [Controllable Dynamic Multi-Task Architectures](https://mlsanity.com/paper/2203.14949/) (2022)
- [Anytime Dense Prediction with Confidence Adaptivity](https://mlsanity.com/paper/2104.00749/) (2021)
- [Task-Customized Self-Supervised Pre-training with Scalable Dynamic Routing](https://mlsanity.com/paper/2205.13267/) (2022)
- [GPFL: Simultaneously Learning Global and Personalized Feature Information for Personalized Federated Learning](https://mlsanity.com/paper/2308.10279/) (2023)
- [AppealNet: An Efficient and Highly-Accurate Edge/Cloud Collaborative Architecture for DNN Inference](https://mlsanity.com/paper/2105.04104/) (2021)
- [DeepPerform: An Efficient Approach for Performance Testing of Resource-Constrained Neural Networks](https://mlsanity.com/paper/2210.05370/) (2022)
- [Towards Inference Efficient Deep Ensemble Learning](https://mlsanity.com/paper/2301.12378/) (2023)
- [DyCL: Dynamic Neural Network Compilation Via Program Rewriting and Graph Optimization](https://mlsanity.com/paper/2307.04963/) (2023)
- [3D Occupancy Prediction with Low-Resolution Queries via Prototype-aware View Transformation](https://mlsanity.com/paper/2503.15185/) (2025)
- [GradMDM: Adversarial Attack on Dynamic Networks](https://mlsanity.com/paper/2304.06724/) (2023)
- [Boost Test-Time Performance with Closed-Loop Inference](https://mlsanity.com/paper/2203.10853/) (2022)
- [Edge Video Analytics: A Survey on Applications, Systems and Enabling Techniques](https://mlsanity.com/paper/2211.15751/) (2022)
- [CBANet: Towards Complexity and Bitrate Adaptive Deep Image Compression using a Single Network](https://mlsanity.com/paper/2105.12386/) (2021)
- [Feature Forgetting in Continual Representation Learning](https://mlsanity.com/paper/2205.13359/) (2022)
- [Flow: Per-Instance Personalized Federated Learning Through Dynamic Routing](https://mlsanity.com/paper/2211.15281/) (2022)
- [Combining Multi-Objective Bayesian Optimization with Reinforcement Learning for TinyML](https://mlsanity.com/paper/2305.14109/) (2023)
- [EREBA: Black-box Energy Testing of Adaptive Neural Networks](https://mlsanity.com/paper/2202.06084/) (2022)
- [A Survey on Dynamic Neural Networks: from Computer Vision to Multi-modal Sensor Fusion](https://mlsanity.com/paper/2501.07451/) (2025)

## Abstract (arXiv)

We introduce Dynamic Deep Neural Networks (D2NN), a new type of feed-forward deep neural network that allows selective execution. Given an input, only a subset of D2NN neurons are executed, and the particular subset is determined by the D2NN itself. By pruning unnecessary computation depending on input, D2NNs provide a way to improve computational efficiency. To achieve dynamic selective execution, a D2NN augments a feed-forward deep neural network (directed acyclic graph of differentiable modules) with controller modules. Each controller module is a sub-network whose output is a decision that controls whether other modules can execute. A D2NN is trained end to end. Both regular and controller modules in a D2NN are learnable and are jointly trained to optimize both accuracy and efficiency. Such training is achieved by integrating backpropagation with reinforcement learning. With extensive experiments of various D2NN architectures on image classification tasks, we demonstrate that D2NNs are general and flexible, and can effectively optimize accuracy-efficiency trade-offs.