# Adaptive Depth in Looped Transformers: Diagnosing Learned Halting Gates and Trajectory Readouts

arXiv: https://arxiv.org/abs/2607.20519 · HTML: https://mlsanity.com/paper/2607.20519/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 32 references that are in the map.
- First posted on arXiv 8 Jul 2026.

## Builds on (most cited first, 30 of 31)

- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge](https://mlsanity.com/paper/1803.05457/) (2018)
- [HellaSwag: Can a Machine Really Finish Your Sentence?](https://mlsanity.com/paper/1905.07830/) (2019)
- [Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity](https://mlsanity.com/paper/2101.03961/) (2021)
- [ALBERT: A Lite BERT for Self-supervised Learning of Language Representations](https://mlsanity.com/paper/1909.11942/) (2019)
- [Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering](https://mlsanity.com/paper/1809.02789/) (2018)
- [CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge](https://mlsanity.com/paper/1811.00937/) (2018)
- [Training Large Language Models to Reason in a Continuous Latent Space](https://mlsanity.com/paper/2412.06769/) (2024)
- [Deep Equilibrium Models](https://mlsanity.com/paper/1909.01377/) (2019)
- [Adaptive Computation Time for Recurrent Neural Networks](https://mlsanity.com/paper/1603.08983/) (2016)
- [Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach](https://mlsanity.com/paper/2502.05171/) (2025)
- [DeeBERT: Dynamic Early Exiting for Accelerating BERT Inference](https://mlsanity.com/paper/2004.12993/) (2020)
- [LayerSkip: Enabling Early Exit Inference and Self-Speculative Decoding](https://mlsanity.com/paper/2404.16710/) (2024)
- [Quiet-STaR: Language Models Can Teach Themselves to Think Before Speaking](https://mlsanity.com/paper/2403.09629/) (2024)
- [BERT Loses Patience: Fast and Robust Inference with Early Exit](https://mlsanity.com/paper/2006.04152/) (2020)
- [Mixture-of-Depths: Dynamically allocating compute in transformer-based language models](https://mlsanity.com/paper/2404.02258/) (2024)
- [DynaBERT: Dynamic BERT with Adaptive Width and Depth](https://mlsanity.com/paper/2004.04037/) (2020)
- [Reasoning with Latent Thoughts: On the Power of Looped Transformers](https://mlsanity.com/paper/2502.17416/) (2025)
- [Looped Transformers as Programmable Computers](https://mlsanity.com/paper/2301.13196/) (2023)
- [Depth-Adaptive Transformer](https://mlsanity.com/paper/1910.10073/) (2019)
- [Exploring Length Generalization in Large Language Models](https://mlsanity.com/paper/2207.04901/) (2022)
- [PonderNet: Learning to Ponder](https://mlsanity.com/paper/2107.05407/) (2021)
- [Scaling Latent Reasoning via Looped Language Models](https://mlsanity.com/paper/2510.25741/) (2025)
- [Looped Transformers are Better at Learning Learning Algorithms](https://mlsanity.com/paper/2311.12424/) (2023)
- [Mixture-of-Recursions: Learning Dynamic Recursive Depths for Adaptive Token-Level Computation](https://mlsanity.com/paper/2507.10524/) (2025)
- [Fast and Robust Early-Exiting Framework for Autoregressive Language Models with Synchronized Parallel Decoding](https://mlsanity.com/paper/2310.05424/) (2023)
- [LoopFormer: Elastic-Depth Looped Transformers for Latent Reasoning via Shortcut Modulation](https://mlsanity.com/paper/2602.11451/) (2026)
- [Loop, Think, & Generalize: Implicit Reasoning in Recurrent-Depth Transformers](https://mlsanity.com/paper/2604.07822/) (2026)
- [Physics of Language Models: Part 4.1, Architecture Design and the Magic of Canon Layers](https://mlsanity.com/paper/2512.17351/) (2025)
- [AdaPonderLM: Gated Pondering Language Models with Token-Wise Adaptive Depth](https://mlsanity.com/paper/2603.01914/) (2026)

## Built on by (most cited first, 12 of 12)

- [The Ignition Is Real, and It Lives at the Readout: Latent composition, difficulty-clocked ignition, and the interface-constituted commit in a recurrent-depth reasoner](https://mlsanity.com/paper/2608.03263/) (2026)
- [Looped Language Models Improve Compositional Tool Calling](https://mlsanity.com/paper/2608.18171/) (2026)
- [Improving Test-Time Scaling with Adaptive Looped Transformers](https://mlsanity.com/paper/2609.35748/) (2026)
- [FlexLoop: Depth-Elastic Looped Policies for Adaptive Test-Time Computation in Deep RL](https://mlsanity.com/paper/2609.34488/) (2026)
- [What Makes Recurrence Effective in Looped Language Models?](https://mlsanity.com/paper/2609.36636/) (2026)
- [LatentHarness: Learning Latent Actions for Memory and Reasoning via Counterfactual Policy Distillation](https://mlsanity.com/paper/2609.39740/) (2026)
- [Shared Weights, Selected Computations: How Looped Transformers Route What Each Loop Does](https://mlsanity.com/paper/2609.39892/) (2026)
- [Towards Looped Models Done Right, Part II: Rethinking at Fixed Points](https://mlsanity.com/paper/2610.06833/) (2026)
- [SanSi: A Looped Typed Decision Model for System 1.5 Thinking](https://mlsanity.com/paper/2610.07730/) (2026)
- [Running Fewer Loops Removes Refusal Behaviour in Recurrent-Depth Language Models](https://mlsanity.com/paper/2610.15218/) (2026)
- [Monitoring latent reasoning in looped language models](https://mlsanity.com/paper/2610.18127/) (2026)
- [Learning to Compute Is Not Learning to Retain or Reuse in Looped Transformers](https://mlsanity.com/paper/2610.19768/) (2026)

## Abstract (arXiv)

Looped Transformers increase test-time computation by repeatedly applying a shared recurrent block. Learned halting objectives in looped Transformers typically use a single exit distribution both as the inference-time stopping rule and as the training-time weighting of per-depth losses. This entangles exit selection with trajectory formation: the gate not only chooses which recurrent state to use, but also determines how strongly each intermediate state is supervised. Consequently, poor adaptive-compute performance can arise from the readout, the induced trajectory, or their interaction. We study adaptive depth in looped Transformers through this trajectory--readout lens, across controlled synthetic tasks (modular arithmetic and binary parity) and large-scale Ouro-1.4B and 2.6B checkpoints. We find that fixed-prior depth supervision, which shapes the trajectory without an input-dependent halting policy, produces difficulty-aware trajectories whose intermediate states expose useful stopping signals, and that simple post-hoc confidence readouts often match or outperform learned linear and MLP gates. Fitting gates on frozen trajectories localizes the failure: it appears to stem mainly from the trajectory induced by joint gate training rather than from limited gate expressivity. The same pattern is present in Ouro evaluations, where pretrained ponder gates are competitive but not uniformly Pareto-optimal, and measured latency confirms that the resulting reductions in average exit depth translate into practical inference-time savings. Our systematic diagnostic evaluation reframes adaptive depth in looped Transformers as a joint problem of trajectory formation and exit readout, rather than gate learning alone, highlighting a distinction that prior learned-halting work has often left implicit.