LayerNorm as Implicit Gain Control in Looped Transformers
Not on the map. Why?
Abstract (arXiv)
In pre-LayerNorm looped transformers, LayerNorm inside the recurrent block acts as an implicit gain controller: by coupling the block's local Lipschitz constant inversely to the activation scale, it renders the recurrence Jacobian non-normal -- asymptotically contractive at every verified fixed point even where its operator norm exceeds 1 -- so the true stability budget is the spectral margin, not an operator-norm bound. That margin depletes as the carry ρ → 1, and a minority of initializations never converge to a fixed point at all, so the diagonal carry constraint ρ(barA) < 1 is necessary but not sufficient for convergence of the full recurrence. Training experiments across six tasks, including a controlled ablation, reveal that the linear carry is not the depth-memory mechanism: gradient descent routes memory through the block's more expressive nonlinear recurrence and leaves the stability-constrained carry at rest -- the carry's role is stabilization, not memory. We characterize the boundary of this claim: on tasks with axis-aligned per-channel structure, gradient descent does recruit the carry. All results are derived analytically and verified in a from-scratch, CPU-scale implementation; verification at larger scale is needed.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 10 of its 11 references that are in the map. First posted on arXiv 12 Jul 2026.
Builds on
10 of 11 references.
References · 10
- 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,718
- 2021Efficiently Modeling Long Sequences with Structured State Spacescited by 2,465
- 2024Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Dualitycited by 1,336
- 2020HiPPO: Recurrent Memory with Optimal Polynomial Projectionscited by 605
- 2016Adaptive Computation Time for Recurrent Neural Networkscited by 506
- 2022On the Parameterization and Initialization of Diagonal State Space Modelscited by 424
- 2025Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approachcited by 415
- 2025Reasoning with Latent Thoughts: On the Power of Looped Transformerscited by 240
- 2026Parcae: Scaling Laws For Stable Looped Language Modelscited by 55
- 2026Loop, Think, & Generalize: Implicit Reasoning in Recurrent-Depth Transformerscited by 37
Built on by
Not cited yet.