# How Much Is One Recurrence Worth? Iso-Depth Scaling Laws for Looped Language Models

arXiv: https://arxiv.org/abs/2604.21106 · HTML: https://mlsanity.com/paper/2604.21106/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- Builds on 41 of its 57 references that are in the map.
- First posted on arXiv 22 Apr 2026.

## Builds on (most cited first, 30 of 41)

- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge](https://mlsanity.com/paper/1803.05457/) (2018)
- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021)
- [Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification](https://mlsanity.com/paper/1502.01852/) (2015)
- [Training Compute-Optimal Large Language Models](https://mlsanity.com/paper/2203.15556/) (2022)
- [TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension](https://mlsanity.com/paper/1705.03551/) (2017)
- [FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning](https://mlsanity.com/paper/2307.08691/) (2023)
- [Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models](https://mlsanity.com/paper/2206.04615/) (2022)
- [Know What You Don't Know: Unanswerable Questions for SQuAD](https://mlsanity.com/paper/1806.03822/) (2018)
- [Root Mean Square Layer Normalization](https://mlsanity.com/paper/1910.07467/) (2019)
- [The LAMBADA dataset: Word prediction requiring a broad discourse context](https://mlsanity.com/paper/1606.06031/) (2016)
- [In-context Learning and Induction Heads](https://mlsanity.com/paper/2209.11895/) (2022)
- [DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphs](https://mlsanity.com/paper/1903.00161/) (2019)
- [Are NLP Models really able to Solve Simple Math Word Problems?](https://mlsanity.com/paper/2103.07191/) (2021)
- [Scaling Vision Transformers to 22 Billion Parameters](https://mlsanity.com/paper/2302.05442/) (2023)
- [CoQA: A Conversational Question Answering Challenge](https://mlsanity.com/paper/1808.07042/) (2018)
- [Universal Transformers](https://mlsanity.com/paper/1807.03819/) (2018)
- [FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precision](https://mlsanity.com/paper/2407.08608/) (2024)
- [TyDi QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse Languages](https://mlsanity.com/paper/2003.05002/) (2020)
- [Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach](https://mlsanity.com/paper/2502.05171/) (2025)
- [DataComp-LM: In search of the next generation of training sets for language models](https://mlsanity.com/paper/2406.11794/) (2024)
- [A Diverse Corpus for Evaluating and Developing English Math Word Problem Solvers](https://mlsanity.com/paper/2106.15772/) (2021)
- [Reasoning with Latent Thoughts: On the Power of Looped Transformers](https://mlsanity.com/paper/2502.17416/) (2025)
- [Looped Transformers as Programmable Computers](https://mlsanity.com/paper/2301.13196/) (2023)
- [Scaling Latent Reasoning via Looped Language Models](https://mlsanity.com/paper/2510.25741/) (2025)
- [Primer: Searching for Efficient Transformers for Language Modeling](https://mlsanity.com/paper/2109.08668/) (2021)
- [Beyond Chinchilla-Optimal: Accounting for Inference in Language Model Scaling Laws](https://mlsanity.com/paper/2401.00448/) (2023)
- [Mixture-of-Recursions: Learning Dynamic Recursive Depths for Adaptive Token-Level Computation](https://mlsanity.com/paper/2507.10524/) (2025)
- [Fantastic Pretraining Optimizers and Where to Find Them](https://mlsanity.com/paper/2509.02046/) (2025)

## Built on by (most cited first, 24 of 24)

- [Equilibrium Reasoners: Learning Attractors Enables Scalable Reasoning](https://mlsanity.com/paper/2605.21488/) (2026)
- [SMELT: Scaling Laws for Compute-Matched MoE Looped Transformers](https://mlsanity.com/paper/2609.01343/) (2026)
- [Hyperloop Transformers](https://mlsanity.com/paper/2604.21254/) (2026)
- [Loop the Loopies!](https://mlsanity.com/paper/2607.16051/) (2026)
- [Depth-adaptive Inference of Looped Language Models via Continuous Depth Batching](https://mlsanity.com/paper/2608.09444/) (2026)
- [Dense Supervision Is Not Enough: The Readout Blind Spot in Looped Language Models](https://mlsanity.com/paper/2606.24898/) (2026)
- [CART: Context-Anchored Recurrent Transformer -- A Parameter-Efficient Architecture with Learned Stability](https://mlsanity.com/paper/2606.01495/) (2026)
- [Tying the Loop -- Tied Expert Layers in Mixture-of-Experts Language Models](https://mlsanity.com/paper/2606.16825/) (2026)
- [LoopCoder-v2: Only Loop Once for Efficient Test-Time Computation Scaling](https://mlsanity.com/paper/2606.18023/) (2026)
- [How Model Growth, Recursion, and Boundary Operators Influence Scaling Exponents](https://mlsanity.com/paper/2609.19107/) (2026)
- [Looped Diffusion Language Models](https://mlsanity.com/paper/2605.26106/) (2026)
- [Improving Test-Time Scaling with Adaptive Looped Transformers](https://mlsanity.com/paper/2609.35748/) (2026)
- [How to Loop MoE: Flatten the Experts, Untie the Attention](https://mlsanity.com/paper/2609.35751/) (2026)
- [Scaling Laws for Looped Mixture of Experts](https://mlsanity.com/paper/2609.40316/) (2026)
- [LiFT: Loop Flow Transformers](https://mlsanity.com/paper/2610.05538/) (2026)
- [Do Language Models Need Sleep? Offline Recurrence for Improved Online Inference](https://mlsanity.com/paper/2605.26099/) (2026)
- [The Model Shape Behind Scaling Laws: A Gradient Superposition Perspective](https://mlsanity.com/paper/2605.27989/) (2026)
- [Learn Your Own Thoughts: Abstract Token Curriculum](https://mlsanity.com/paper/2609.19717/) (2026)
- [FluidRain: Incompressible Rain Flow as an Attention Bias for Loop-in-Loop Video Deraining](https://mlsanity.com/paper/2609.29006/) (2026)
- [What Makes Recurrence Effective in Looped Language Models?](https://mlsanity.com/paper/2609.36636/) (2026)
- [LoopVL: Recurrent Visual Intelligence](https://mlsanity.com/paper/2609.38426/) (2026)
- [Depth as Time in One-Step Generative Models](https://mlsanity.com/paper/2610.03626/) (2026)
- [One Model to Loop Them All: Elastic Latent Recurrence for Language Models](https://mlsanity.com/paper/2610.12582/) (2026)
- [Rotate the Loop: Blockwise Conformality for Looped Transformers](https://mlsanity.com/paper/2610.14030/) (2026)

## Abstract (arXiv)

We measure how much one recurrence is worth to a looped (depth-recurrent) transformer, in equivalent unique parameters. From an iso-depth pretraining sweep across recurrence counts r ∈ 1, 2, 4, 8 spanning ~50× in training compute, we fit a joint scaling law L = E + A,(Nonce + rvarphi Nrec)-α + B,D-β and measure a recurrence-equivalence exponent varphi = 0.46. Intuitively, varphi tells us whether looping a block r times is equivalent in validation loss to r unique blocks of a non-looped model (full equivalence, varphi=1) or to a single block run repeatedly with no capacity gain (varphi=0). Our varphi = 0.46 sits in between, so replacing unique blocks with shared recurrences increases validation loss at matched training compute. For example, at r=4 a 410M looped model performs on par with a 580M non-looped model, but incurs the training cost of a 1B non-looped one. We demonstrate the utility of varphi as a diagnostic tool on two case studies: commonly used truncated backpropagation lowers varphi to 0.38, indicating that the loop mechanism is poorly trained under truncation, even though validation loss decreases. Conversely, hyperconnections raise varphi to 0.65, a genuine capacity gain. Our method separates true loop improvements from training-side gains, a distinction raw validation loss cannot make.