# Path Independent Equilibrium Models Can Better Exploit Test-Time Computation

arXiv: https://arxiv.org/abs/2211.09961 · HTML: https://mlsanity.com/paper/2211.09961/

## Key facts

- Cited by 38 papers in the map (data through 14 Oct 2026).
- Builds on 43 of its 60 references that are in the map.
- First posted on arXiv 18 Nov 2022.

## Builds on (most cited first, 30 of 43)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Group Normalization](https://mlsanity.com/paper/1803.08494/) (2018)
- [WILDS: A Benchmark of in-the-Wild Distribution Shifts](https://mlsanity.com/paper/2012.07421/) (2020)
- [Strategies for Pre-training Graph Neural Networks](https://mlsanity.com/paper/1905.12265/) (2019)
- [Show Your Work: Scratchpads for Intermediate Computation with Language Models](https://mlsanity.com/paper/2112.00114/) (2021)
- [Universal Transformers](https://mlsanity.com/paper/1807.03819/) (2018)
- [Deep Equilibrium Models](https://mlsanity.com/paper/1909.01377/) (2019)
- [BranchyNet: Fast Inference via Early Exiting from Deep Neural Networks](https://mlsanity.com/paper/1709.01686/) (2017)
- [Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks](https://mlsanity.com/paper/1602.07868/) (2016)
- [Adaptive Computation Time for Recurrent Neural Networks](https://mlsanity.com/paper/1603.08983/) (2016)
- [SkipNet: Learning Dynamic Routing in Convolutional Networks](https://mlsanity.com/paper/1711.09485/) (2017)
- [BERT Loses Patience: Fast and Robust Inference with Early Exit](https://mlsanity.com/paper/2006.04152/) (2020)
- [Accuracy on the Line: On the Strong Correlation Between Out-of-Distribution and In-Distribution Generalization](https://mlsanity.com/paper/2107.04649/) (2021)
- [Depth-Adaptive Transformer](https://mlsanity.com/paper/1910.10073/) (2019)
- [FastBERT: a Self-distilling BERT with Adaptive Inference Time](https://mlsanity.com/paper/2004.02178/) (2020)
- [Learning a SAT Solver from Single-Bit Supervision](https://mlsanity.com/paper/1802.03685/) (2018)
- [PonderNet: Learning to Ponder](https://mlsanity.com/paper/2107.05407/) (2021)
- [Multiscale Deep Equilibrium Models](https://mlsanity.com/paper/2006.08656/) (2020)
- [The Right Tool for the Job: Matching Model and Instance Complexities](https://mlsanity.com/paper/2004.07453/) (2020)
- [Spatially Adaptive Computation Time for Residual Networks](https://mlsanity.com/paper/1612.02297/) (2016)
- [Can You Learn an Algorithm? Generalizing from Easy to Hard Problems with Recurrent Networks](https://mlsanity.com/paper/2106.04537/) (2021)
- [Monotone operator equilibrium networks](https://mlsanity.com/paper/2006.08591/) (2020)
- [Implicit Graph Neural Networks](https://mlsanity.com/paper/2009.06211/) (2020)
- [Is Attention Better Than Matrix Decomposition?](https://mlsanity.com/paper/2109.04553/) (2021)
- [Adaptive Inference through Early-Exit Networks: Design, Challenges and Directions](https://mlsanity.com/paper/2106.05022/) (2021)
- [On Training Implicit Models](https://mlsanity.com/paper/2111.05177/) (2021)
- [Stabilizing Equilibrium Models by Jacobian Regularization](https://mlsanity.com/paper/2106.14342/) (2021)

## Built on by (most cited first, 30 of 38)

- [Large Language Models Are Zero-Shot Time Series Forecasters](https://mlsanity.com/paper/2310.07820/) (2023)
- [Transformers Can Do Arithmetic with the Right Embeddings](https://mlsanity.com/paper/2405.17399/) (2024)
- [One-Step Diffusion Distillation via Deep Equilibrium Models](https://mlsanity.com/paper/2401.08639/) (2023)
- [Parcae: Scaling Laws For Stable Looped Language Models](https://mlsanity.com/paper/2604.12946/) (2026)
- [Learning Iterative Reasoning through Energy Diffusion](https://mlsanity.com/paper/2406.11179/) (2024)
- [Fixed-Point Reasoners: Stable and Adaptive Deep Looped Transformers](https://mlsanity.com/paper/2606.18206/) (2026)
- [Implicit Language Models are RNNs: Balancing Parallelization and Expressivity](https://mlsanity.com/paper/2502.07827/) (2025)
- [TorchDEQ: A Library for Deep Equilibrium Models](https://mlsanity.com/paper/2310.18605/) (2023)
- [Deep Equilibrium Based Neural Operators for Steady-State PDEs](https://mlsanity.com/paper/2312.00234/) (2023)
- [Equilibrium Reasoners: Learning Attractors Enables Scalable Reasoning](https://mlsanity.com/paper/2605.21488/) (2026)
- [Learning Regularization Functionals for Inverse Problems: A Comparative Study](https://mlsanity.com/paper/2510.01755/) (2025)
- [ELT: Elastic Looped Transformers for Visual Generation](https://mlsanity.com/paper/2604.09168/) (2026)
- [Stability and Generalization in Looped Transformers](https://mlsanity.com/paper/2604.15259/) (2026)
- [Thinking with Looped Flows](https://mlsanity.com/paper/2609.11801/) (2026)
- [On Logical Extrapolation for Mazes with Recurrent and Implicit Networks](https://mlsanity.com/paper/2410.03020/) (2024)
- [AbbIE: Autoregressive Block-Based Iterative Encoder for Efficient Sequence Modeling](https://mlsanity.com/paper/2507.08567/) (2025)
- [Efficient Parallel Samplers for Recurrent-Depth Models and Their Connection to Diffusion Language Models](https://mlsanity.com/paper/2510.14961/) (2025)
- [DEALing with Image Reconstruction: Deep Attentive Least Squares](https://mlsanity.com/paper/2502.04079/) (2025)
- [Simply Stabilizing the Loop via Fully Looped Transformer](https://mlsanity.com/paper/2605.18797/) (2026)
- [When Does Recurrence Become an Algorithm? Convergence Selection in Weight-Tied Looped Transformers](https://mlsanity.com/paper/2607.20594/) (2026)
- [A Generalization Bound for a Family of Implicit Networks](https://mlsanity.com/paper/2410.07427/) (2024)
- [Stabilizing Extrapolation in Looped Transformers via Learned Stochastic Stopping](https://mlsanity.com/paper/2606.29983/) (2026)
- [Test like you Train in Implicit Deep Learning](https://mlsanity.com/paper/2305.15042/) (2023)
- [Iteratively Refined Image Reconstruction with Learned Attentive Regularizers](https://mlsanity.com/paper/2407.06608/) (2024)
- [Accelerating Hopfield Network Dynamics: Beyond Synchronous Updates and Forward Euler](https://mlsanity.com/paper/2311.15673/) (2023)
- [Consistency Deep Equilibrium Models](https://mlsanity.com/paper/2602.03024/) (2026)
- [C-voting: Confidence-Based Test-Time Voting without Explicit Energy Functions](https://mlsanity.com/paper/2604.13521/) (2026)
- [Looped Diffusion Language Models](https://mlsanity.com/paper/2605.26106/) (2026)
- [When Recursive Models Finish Computing](https://mlsanity.com/paper/2609.26487/) (2026)
- [The Extrapolation Power of Implicit Models](https://mlsanity.com/paper/2407.14430/) (2024)

## Abstract (arXiv)

Designing networks capable of attaining better performance with an increased inference budget is important to facilitate generalization to harder problem instances. Recent efforts have shown promising results in this direction by making use of depth-wise recurrent networks. We show that a broad class of architectures named equilibrium models display strong upwards generalization, and find that stronger performance on harder examples (which require more iterations of inference to get correct) strongly correlates with the path independence of the system -- its tendency to converge to the same steady-state behaviour regardless of initialization, given enough computation. Experimental interventions made to promote path independence result in improved generalization on harder problem instances, while those that penalize it degrade this ability. Path independence analyses are also useful on a per-example basis: for equilibrium models that have good in-distribution performance, path independence on out-of-distribution samples strongly correlates with accuracy. Our results help explain why equilibrium models are capable of strong upwards generalization and motivates future work that harnesses path independence as a general modelling principle to facilitate scalable test-time usage.