# Anytime Dense Prediction with Confidence Adaptivity

arXiv: https://arxiv.org/abs/2104.00749 · HTML: https://mlsanity.com/paper/2104.00749/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Builds on 41 of its 55 references that are in the map.
- First posted on arXiv 1 Apr 2021.

## Builds on (most cited first, 30 of 41)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Fully Convolutional Networks for Semantic Segmentation](https://mlsanity.com/paper/1411.4038/) (2014)
- [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018)
- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [Categorical Reparameterization with Gumbel-Softmax](https://mlsanity.com/paper/1611.01144/) (2016)
- [Pyramid Scene Parsing Network](https://mlsanity.com/paper/1612.01105/) (2016)
- [Learning both Weights and Connections for Efficient Neural Networks](https://mlsanity.com/paper/1506.02626/) (2015)
- [FitNets: Hints for Thin Deep Nets](https://mlsanity.com/paper/1412.6550/) (2014)
- [Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference](https://mlsanity.com/paper/1712.05877/) (2017)
- [Deep High-Resolution Representation Learning for Visual Recognition](https://mlsanity.com/paper/1908.07919/) (2019)
- [Deep High-Resolution Representation Learning for Human Pose Estimation](https://mlsanity.com/paper/1902.09212/) (2019)
- [4D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networks](https://mlsanity.com/paper/1904.08755/) (2019)
- [Pruning Filters for Efficient ConvNets](https://mlsanity.com/paper/1608.08710/) (2016)
- [XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks](https://mlsanity.com/paper/1603.05279/) (2016)
- [Rethinking the Value of Network Pruning](https://mlsanity.com/paper/1810.05270/) (2018)
- [BranchyNet: Fast Inference via Early Exiting from Deep Neural Networks](https://mlsanity.com/paper/1709.01686/) (2017)
- [PointRend: Image Segmentation as Rendering](https://mlsanity.com/paper/1912.08193/) (2019)
- [SkipNet: Learning Dynamic Routing in Convolutional Networks](https://mlsanity.com/paper/1711.09485/) (2017)
- [ICNet for Real-Time Semantic Segmentation on High-Resolution Images](https://mlsanity.com/paper/1704.08545/) (2017)
- [cuDNN: Efficient Primitives for Deep Learning](https://mlsanity.com/paper/1410.0759/) (2014)
- [Submanifold Sparse Convolutional Networks](https://mlsanity.com/paper/1706.01307/) (2017)
- [BlockDrop: Dynamic Inference Paths in Residual Networks](https://mlsanity.com/paper/1711.08393/) (2017)
- [The Hardware Lottery](https://mlsanity.com/paper/2009.06489/) (2020)
- [Multiscale Deep Equilibrium Models](https://mlsanity.com/paper/2006.08656/) (2020)
- [Convolutional Networks with Adaptive Inference Graphs](https://mlsanity.com/paper/1711.11503/) (2017)
- [Spatially Adaptive Computation Time for Residual Networks](https://mlsanity.com/paper/1612.02297/) (2016)
- [Dynamic Convolutions: Exploiting Spatial Sparsity for Faster Inference](https://mlsanity.com/paper/1912.03203/) (2019)
- [Fast Sparse ConvNets](https://mlsanity.com/paper/1911.09723/) (2019)

## Built on by (most cited first, 18 of 18)

- [LightGlue: Local Feature Matching at Light Speed](https://mlsanity.com/paper/2306.13643/) (2023)
- [Confident Adaptive Language Modeling](https://mlsanity.com/paper/2207.07061/) (2022)
- [Mixture-of-Depths: Dynamically allocating compute in transformer-based language models](https://mlsanity.com/paper/2404.02258/) (2024)
- [AI Flow: Perspectives, Scenarios, and Approaches](https://mlsanity.com/paper/2506.12479/) (2025)
- [Fast yet Safe: Early-Exiting with Risk Control](https://mlsanity.com/paper/2405.20915/) (2024)
- [UMC: A Unified Bandwidth-efficient and Multi-resolution based Collaborative Perception Framework](https://mlsanity.com/paper/2303.12400/) (2023)
- [Accelerating Inference in Large Language Models with a Unified Layer Skipping Strategy](https://mlsanity.com/paper/2404.06954/) (2024)
- [Towards Anytime Classification in Early-Exit Architectures by Enforcing Conditional Monotonicity](https://mlsanity.com/paper/2306.02652/) (2023)
- [Skip a Layer or Loop it? Test-Time Depth Adaptation of Pretrained LLMs](https://mlsanity.com/paper/2507.07996/) (2025)
- [Adaptive Deep Neural Network Inference Optimization with EENet](https://mlsanity.com/paper/2301.07099/) (2023)
- [StereoVoxelNet: Real-Time Obstacle Detection Based on Occupancy Voxels from a Stereo Camera Using Deep Neural Networks](https://mlsanity.com/paper/2209.08459/) (2022)
- [Class Based Thresholding in Early Exit Semantic Segmentation Networks](https://mlsanity.com/paper/2210.15621/) (2022)
- [Securing Multi-turn Conversational Language Models From Distributed Backdoor Triggers](https://mlsanity.com/paper/2407.04151/) (2024)
- [Understanding the Robustness of Multi-Exit Models under Common Corruptions](https://mlsanity.com/paper/2212.01562/) (2022)
- [Progressive Confident Masking Attention Network for Audio-Visual Segmentation](https://mlsanity.com/paper/2406.02345/) (2024)
- [ElastiFormer: Learned Redundancy Reduction in Transformer via Self-Distillation](https://mlsanity.com/paper/2411.15281/) (2024)
- [Cascading Refinement Video Denoising with Uncertainty Adaptivity](https://mlsanity.com/paper/2408.02284/) (2024)
- [FTP: A Fine-grained Token-wise Pruner for Large Language Models via Token Routing](https://mlsanity.com/paper/2412.11494/) (2024)

## Abstract (arXiv)

Anytime inference requires a model to make a progression of predictions which might be halted at any time. Prior research on anytime visual recognition has mostly focused on image classification. We propose the first unified and end-to-end approach for anytime dense prediction. A cascade of "exits" is attached to the model to make multiple predictions. We redesign the exits to account for the depth and spatial resolution of the features for each exit. To reduce total computation, and make full use of prior predictions, we develop a novel spatially adaptive approach to avoid further computation on regions where early predictions are already sufficiently confident. Our full method, named anytime dense prediction with confidence (ADP-C), achieves the same level of final accuracy as the base model, and meanwhile significantly reduces total computation. We evaluate our method on Cityscapes semantic segmentation and MPII human pose estimation: ADP-C enables anytime inference without sacrificing accuracy while also reducing the total FLOPs of its base models by 44.4% and 59.1%. We compare with anytime inference by deep equilibrium networks and feature-based stochastic sampling, showing that ADP-C dominates both across the accuracy-computation curve. Our code is available at https://github.com/liuzhuang13/anytime .