# Fully Dynamic Inference with Deep Neural Networks

arXiv: https://arxiv.org/abs/2007.15151 · HTML: https://mlsanity.com/paper/2007.15151/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 Jul 2020.

## Built on by (most cited first, 19 of 19)

- [Dynamic Neural Networks: A Survey](https://mlsanity.com/paper/2102.04906/) (2021)
- [AdaViT: Adaptive Tokens for Efficient Vision Transformer](https://mlsanity.com/paper/2112.07658/) (2021)
- [MADTP: Multimodal Alignment-Guided Dynamic Token Pruning for Accelerating Vision-Language Transformer](https://mlsanity.com/paper/2403.02991/) (2024)
- [Structural Pruning via Latency-Saliency Knapsack](https://mlsanity.com/paper/2210.06659/) (2022)
- [Continual Object Detection via Prototypical Task Correlation Guided Gating Mechanism](https://mlsanity.com/paper/2205.03055/) (2022)
- [Referring Expression Comprehension Using Language Adaptive Inference](https://mlsanity.com/paper/2306.04451/) (2023)
- [Adaptive Computation Modules: Granular Conditional Computation For Efficient Inference](https://mlsanity.com/paper/2312.10193/) (2023)
- [CODEBench: A Neural Architecture and Hardware Accelerator Co-Design Framework](https://mlsanity.com/paper/2212.03965/) (2022)
- [Progressive Channel-Shrinking Network](https://mlsanity.com/paper/2304.00280/) (2023)
- [Input Conditioned Layer Dropping in Speech Foundation Models](https://mlsanity.com/paper/2507.07954/) (2025)
- [Predicting Probabilities of Error to Combine Quantization and Early Exiting: QuEE](https://mlsanity.com/paper/2406.14404/) (2024)
- [SCAI: A Spectral data Classification framework with Adaptive Inference for the IoT platform](https://mlsanity.com/paper/2206.12420/) (2022)
- [High-Level Parallelism and Nested Features for Dynamic Inference Cost and Top-Down Attention](https://mlsanity.com/paper/2308.05128/) (2023)
- [Successive Refinement in Large-Scale Computation: Advancing Model Inference Applications](https://mlsanity.com/paper/2402.07229/) (2024)
- [Model Adaptation for Time Constrained Embodied Control](https://mlsanity.com/paper/2406.11128/) (2024)
- [MIGT: Memory Instance Gated Transformer Framework for Financial Portfolio Management](https://mlsanity.com/paper/2502.07280/) (2025)
- [Training-Free Inference for High-Resolution Sinogram Completion](https://mlsanity.com/paper/2506.08809/) (2025)
- [A Vision Toward Energy-Efficient Domain-Specific Artificial Intelligence Models and Agents](https://mlsanity.com/paper/2510.22052/) (2025)
- [DANCE: Dynamic 3D CNN Pruning: Joint Frame, Channel, and Feature Adaptation for Energy Efficiency on the Edge](https://mlsanity.com/paper/2603.17275/) (2026)

## Abstract (arXiv)

Modern deep neural networks are powerful and widely applicable models that extract task-relevant information through multi-level abstraction. Their cross-domain success, however, is often achieved at the expense of computational cost, high memory bandwidth, and long inference latency, which prevents their deployment in resource-constrained and time-sensitive scenarios, such as edge-side inference and self-driving cars. While recently developed methods for creating efficient deep neural networks are making their real-world deployment more feasible by reducing model size, they do not fully exploit input properties on a per-instance basis to maximize computational efficiency and task accuracy. In particular, most existing methods typically use a one-size-fits-all approach that identically processes all inputs. Motivated by the fact that different images require different feature embeddings to be accurately classified, we propose a fully dynamic paradigm that imparts deep convolutional neural networks with hierarchical inference dynamics at the level of layers and individual convolutional filters/channels. Two compact networks, called Layer-Net (L-Net) and Channel-Net (C-Net), predict on a per-instance basis which layers or filters/channels are redundant and therefore should be skipped. L-Net and C-Net also learn how to scale retained computation outputs to maximize task accuracy. By integrating L-Net and C-Net into a joint design framework, called LC-Net, we consistently outperform state-of-the-art dynamic frameworks with respect to both efficiency and classification accuracy. On the CIFAR-10 dataset, LC-Net results in up to 11.9× fewer floating-point operations (FLOPs) and up to 3.3% higher accuracy compared to other dynamic inference methods. On the ImageNet dataset, LC-Net achieves up to 1.4× fewer FLOPs and up to 4.6% higher Top-1 accuracy than the other methods.