# Adaptive recurrent vision performs zero-shot computation scaling to unseen difficulty levels

arXiv: https://arxiv.org/abs/2311.06964 · HTML: https://mlsanity.com/paper/2311.06964/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 29 references that are in the map.
- First posted on arXiv 12 Nov 2023.

## Builds on (most cited first, 15 of 15)

- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Long Range Arena: A Benchmark for Efficient Transformers](https://mlsanity.com/paper/2011.04006/) (2020)
- [Adaptive Computation Time for Recurrent Neural Networks](https://mlsanity.com/paper/1603.08983/) (2016)
- [PonderNet: Learning to Ponder](https://mlsanity.com/paper/2107.05407/) (2021)
- [Delving Deeper into Convolutional Networks for Learning Video Representations](https://mlsanity.com/paper/1511.06432/) (2015)
- [Can You Learn an Algorithm? Generalizing from Easy to Hard Problems with Recurrent Networks](https://mlsanity.com/paper/2106.04537/) (2021)
- [Learning long-range spatial dependencies with horizontal gated-recurrent units](https://mlsanity.com/paper/1805.08315/) (2018)
- [Stable Recurrent Models](https://mlsanity.com/paper/1805.10369/) (2018)
- [Bridging the Gaps Between Residual Learning, Recurrent Neural Networks and Visual Cortex](https://mlsanity.com/paper/1604.03640/) (2016)
- [End-to-end Algorithm Synthesis with Recurrent Networks: Logical Extrapolation Without Overthinking](https://mlsanity.com/paper/2202.05826/) (2022)
- [Stable and expressive recurrent vision models](https://mlsanity.com/paper/2005.11362/) (2020)
- [Differentiable Adaptive Computation Time for Visual Reasoning](https://mlsanity.com/paper/2004.12770/) (2020)
- [Understanding Deep Architectures using a Recursive Convolutional Network](https://mlsanity.com/paper/1312.1847/) (2013)
- [The Uncanny Similarity of Recurrence and Depth](https://mlsanity.com/paper/2102.11011/) (2021)
- [Improving Anytime Prediction with Parallel Cascaded Networks and a Temporal-Difference Loss](https://mlsanity.com/paper/2102.09808/) (2021)

## Built on by (most cited first, 6 of 6)

- [Understanding and Improving Length Generalization in Recurrent Models](https://mlsanity.com/paper/2507.02782/) (2025)
- [Learning to Stop Overthinking at Test Time](https://mlsanity.com/paper/2502.10954/) (2025)
- [NeuralSolver: Learning Algorithms For Consistent and Efficient Extrapolation Across General Tasks](https://mlsanity.com/paper/2402.15393/) (2024)
- [When in Doubt, Think Slow: Iterative Reasoning with Latent Imagination](https://mlsanity.com/paper/2402.15283/) (2024)
- [Learning Model Successors](https://mlsanity.com/paper/2502.00197/) (2025)
- [DART: Distributional Adversarial Recurrent Training for Algorithm Learning](https://mlsanity.com/paper/2609.05988/) (2026)

## Abstract (arXiv)

Humans solving algorithmic (or) reasoning problems typically exhibit solution times that grow as a function of problem difficulty. Adaptive recurrent neural networks have been shown to exhibit this property for various language-processing tasks. However, little work has been performed to assess whether such adaptive computation can also enable vision models to extrapolate solutions beyond their training distribution's difficulty level, with prior work focusing on very simple tasks. In this study, we investigate a critical functional role of such adaptive processing using recurrent neural networks: to dynamically scale computational resources conditional on input requirements that allow for zero-shot generalization to novel difficulty levels not seen during training using two challenging visual reasoning tasks: PathFinder and Mazes. We combine convolutional recurrent neural networks (ConvRNNs) with a learnable halting mechanism based on Graves (2016). We explore various implementations of such adaptive ConvRNNs (AdRNNs) ranging from tying weights across layers to more sophisticated biologically inspired recurrent networks that possess lateral connections and gating. We show that 1) AdRNNs learn to dynamically halt processing early (or late) to solve easier (or harder) problems, 2) these RNNs zero-shot generalize to more difficult problem settings not shown during training by dynamically increasing the number of recurrent iterations at test time. Our study provides modeling evidence supporting the hypothesis that recurrent processing enables the functional advantage of adaptively allocating compute resources conditional on input requirements and hence allowing generalization to harder difficulty levels of a visual reasoning problem without training.