# Dense Supervision Is Not Enough: The Readout Blind Spot in Looped Language Models

arXiv: https://arxiv.org/abs/2606.24898 · HTML: https://mlsanity.com/paper/2606.24898/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 27 references that are in the map.
- First posted on arXiv 12 Jun 2026.

## Builds on (most cited first, 26 of 26)

- [Pointer Sentinel Mixture Models](https://mlsanity.com/paper/1609.07843/) (2016)
- [Root Mean Square Layer Normalization](https://mlsanity.com/paper/1910.07467/) (2019)
- [The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale](https://mlsanity.com/paper/2406.17557/) (2024)
- [On Layer Normalization in the Transformer Architecture](https://mlsanity.com/paper/2002.04745/) (2020)
- [Universal Transformers](https://mlsanity.com/paper/1807.03819/) (2018)
- [Adaptive Computation Time for Recurrent Neural Networks](https://mlsanity.com/paper/1603.08983/) (2016)
- [Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach](https://mlsanity.com/paper/2502.05171/) (2025)
- [Reasoning with Latent Thoughts: On the Power of Looped Transformers](https://mlsanity.com/paper/2502.17416/) (2025)
- [Depth-Adaptive Transformer](https://mlsanity.com/paper/1910.10073/) (2019)
- [DeepNet: Scaling Transformers to 1,000 Layers](https://mlsanity.com/paper/2203.00555/) (2022)
- [PonderNet: Learning to Ponder](https://mlsanity.com/paper/2107.05407/) (2021)
- [Scaling Latent Reasoning via Looped Language Models](https://mlsanity.com/paper/2510.25741/) (2025)
- [NormFormer: Improved Transformer Pretraining with Extra Normalization](https://mlsanity.com/paper/2110.09456/) (2021)
- [LoopFormer: Elastic-Depth Looped Transformers for Latent Reasoning via Shortcut Modulation](https://mlsanity.com/paper/2602.11451/) (2026)
- [Parcae: Scaling Laws For Stable Looped Language Models](https://mlsanity.com/paper/2604.12946/) (2026)
- [Loop, Think, & Generalize: Implicit Reasoning in Recurrent-Depth Transformers](https://mlsanity.com/paper/2604.07822/) (2026)
- [A Mechanistic Analysis of Looped Reasoning Language Models](https://mlsanity.com/paper/2604.11791/) (2026)
- [How Much Is One Recurrence Worth? Iso-Depth Scaling Laws for Looped Language Models](https://mlsanity.com/paper/2604.21106/) (2026)
- [Stabilizing Recurrent Dynamics for Test-Time Scalable Latent Reasoning in Looped Language Models](https://mlsanity.com/paper/2605.26733/) (2026)
- [ELT: Elastic Looped Transformers for Visual Generation](https://mlsanity.com/paper/2604.09168/) (2026)
- [Sparse Layers are Critical to Scaling Looped Language Models](https://mlsanity.com/paper/2605.09165/) (2026)
- [LoopRPT: Reinforcement Pre-Training for Looped Language Models](https://mlsanity.com/paper/2603.19714/) (2026)
- [Stability and Generalization in Looped Transformers](https://mlsanity.com/paper/2604.15259/) (2026)
- [Prioritize the Process, Not Just the Outcome: Rewarding Latent Thought Trajectories Improves Reasoning in Looped Language Models](https://mlsanity.com/paper/2602.10520/) (2026)
- [Simply Stabilizing the Loop via Fully Looped Transformer](https://mlsanity.com/paper/2605.18797/) (2026)
- [Gated Normalization Removal and Scale Anchoring in Pre-Norm Transformers](https://mlsanity.com/paper/2602.10408/) (2026)

## Built on by (most cited first, 5 of 5)

- [Full-bandwidth transformer](https://mlsanity.com/paper/2608.08888/) (2026)
- [The Ignition Is Real, and It Lives at the Readout: Latent composition, difficulty-clocked ignition, and the interface-constituted commit in a recurrent-depth reasoner](https://mlsanity.com/paper/2608.03263/) (2026)
- [Improving Test-Time Scaling with Adaptive Looped Transformers](https://mlsanity.com/paper/2609.35748/) (2026)
- [Recursive Vision Language Models for General Symbolic Reasoning](https://mlsanity.com/paper/2608.01534/) (2026)
- [Shared Weights, Selected Computations: How Looped Transformers Route What Each Loop Does](https://mlsanity.com/paper/2609.39892/) (2026)

## Abstract (arXiv)

Looped language models turn hidden states into runtime state: each state is decoded for prediction and fed back into future computation. This creates a basic supervision question: which state variables does cross-entropy actually control? We show that dense per-loop cross-entropy controls the variables exposed by the readout, not every variable active in the recurrent transition. Hidden-state scale gives a concrete failure mode. Scale-invariant readouts such as RMSNorm and LayerNorm hide radial scale from the immediate cross-entropy loss, while pre-norm residual recurrence continues to carry and update that same scale. Thus per-loop loss can make early exits usable without controlling recurrent scale. In 44M and 129M looped transformers without inter-loop normalization, per-loop cross-entropy through RMSNorm readouts still drives final hidden-state norms into the thousands or tens of thousands. Scale-visible readouts and explicit norm penalties keep norms in the tens, and scale-removing recurrence is the complementary architectural fix. The resulting design rule is simple: dense supervision trains exits; recurrent scale control requires either making scale visible to a loss or removing it from the loop. Consistent with this rule, scale-controlled variants achieve lower perplexity at matched inference-depth operating points in our variable-depth benchmarks.