# Improving Anytime Prediction with Parallel Cascaded Networks and a Temporal-Difference Loss

arXiv: https://arxiv.org/abs/2102.09808 · HTML: https://mlsanity.com/paper/2102.09808/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 65 references that are in the map.
- First posted on arXiv 19 Feb 2021.

## Builds on (most cited first, 30 of 31)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [Wide Residual Networks](https://mlsanity.com/paper/1605.07146/) (2016)
- [LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loop](https://mlsanity.com/paper/1506.03365/) (2015)
- [Enhancing The Reliability of Out-of-distribution Image Detection in Neural Networks](https://mlsanity.com/paper/1706.02690/) (2017)
- [Stacked Hourglass Networks for Human Pose Estimation](https://mlsanity.com/paper/1603.06937/) (2016)
- [BranchyNet: Fast Inference via Early Exiting from Deep Neural Networks](https://mlsanity.com/paper/1709.01686/) (2017)
- [How far are we from solving the 2D & 3D Face Alignment problem? (and a dataset of 230,000 3D facial landmarks)](https://mlsanity.com/paper/1703.07332/) (2017)
- [Adaptive Computation Time for Recurrent Neural Networks](https://mlsanity.com/paper/1603.08983/) (2016)
- [Highway Networks](https://mlsanity.com/paper/1505.00387/) (2015)
- [Depth-Adaptive Transformer](https://mlsanity.com/paper/1910.10073/) (2019)
- [Human uncertainty makes classification more robust](https://mlsanity.com/paper/1908.07086/) (2019)
- [FractalNet: Ultra-Deep Neural Networks without Residuals](https://mlsanity.com/paper/1605.07648/) (2016)
- [Adaptive Neural Networks for Efficient Inference](https://mlsanity.com/paper/1702.07811/) (2017)
- [Resolution Adaptive Networks for Efficient Inference](https://mlsanity.com/paper/2003.07326/) (2020)
- [Characterizing Structural Regularities of Labeled Data in Overparameterized Models](https://mlsanity.com/paper/2002.03206/) (2020)
- [Why should we add early exits to neural networks?](https://mlsanity.com/paper/2004.12814/) (2020)
- [IDK Cascades: Fast Deep Learning by Learning not to Overthink](https://mlsanity.com/paper/1706.00885/) (2017)
- [Learning to Stop While Learning to Predict](https://mlsanity.com/paper/2006.05082/) (2020)
- [Feedback Networks](https://mlsanity.com/paper/1612.09508/) (2016)
- [Toward fast and accurate human pose estimation via soft-gated skip connections](https://mlsanity.com/paper/2002.11098/) (2020)
- [Massively Parallel Video Networks](https://mlsanity.com/paper/1806.03863/) (2018)
- [IamNN: Iterative and Adaptive Mobile Neural Network for Efficient Image Classification](https://mlsanity.com/paper/1804.10123/) (2018)
- [Anytime Neural Prediction via Slicing Networks Vertically](https://mlsanity.com/paper/1807.02609/) (2018)
- [The streaming rollout of deep networks - towards fully model-parallel execution](https://mlsanity.com/paper/1806.04965/) (2018)
- [Anytime Prediction as a Model of Human Reaction Time](https://mlsanity.com/paper/2011.12859/) (2020)
- [Recurrent Segmentation for Variable Computational Budgets](https://mlsanity.com/paper/1711.10151/) (2017)

## Built on by (most cited first, 6 of 6)

- [Handling Delay in Real-Time Reinforcement Learning](https://mlsanity.com/paper/2503.23478/) (2025)
- [Adaptive recurrent vision performs zero-shot computation scaling to unseen difficulty levels](https://mlsanity.com/paper/2311.06964/) (2023)
- [Understanding the Robustness of Multi-Exit Models under Common Corruptions](https://mlsanity.com/paper/2212.01562/) (2022)
- [Signatures of human-like processing in Transformer forward passes](https://mlsanity.com/paper/2504.14107/) (2025)
- [Catch Your Breath: Adaptive Computation for Self-Paced Sequence Production](https://mlsanity.com/paper/2510.13879/) (2025)
- [SATBench: Benchmarking the speed-accuracy tradeoff in object recognition by humans and dynamic neural networks](https://mlsanity.com/paper/2206.08427/) (2022)

## Abstract (arXiv)

Although deep feedforward neural networks share some characteristics with the primate visual system, a key distinction is their dynamics. Deep nets typically operate in serial stages wherein each layer completes its computation before processing begins in subsequent layers. In contrast, biological systems have cascaded dynamics: information propagates from neurons at all layers in parallel but transmission occurs gradually over time, leading to speed-accuracy trade offs even in feedforward architectures. We explore the consequences of biologically inspired parallel hardware by constructing cascaded ResNets in which each residual block has propagation delays but all blocks update in parallel in a stateful manner. Because information transmitted through skip connections avoids delays, the functional depth of the architecture increases over time, yielding anytime predictions that improve with internal-processing time. We introduce a temporal-difference training loss that achieves a strictly superior speed-accuracy profile over standard losses and enables the cascaded architecture to outperform state-of-the-art anytime-prediction methods. The cascaded architecture has intriguing properties, including: it classifies typical instances more rapidly than atypical instances; it is more robust to both persistent and transient noise than is a conventional ResNet; and its time-varying output trace provides a signal that can be exploited to improve information processing and inference.