mlsanityData to 14 Oct 2026

When Does Recurrence Become an Algorithm? Convergence Selection in Weight-Tied Looped Transformers

arXiv 2607.20594 ↗First posted 22 Jul 2026cs.LG cs.AI stat.MLcited by 6

Not on the map. Why?

Abstract (arXiv)

When does a weight-tied looped transformer -- one block applied T times -- implement an actual algorithm? We answer with four findings from controlled populations on group word problems. (1) The budget law: free training installs a linear computation frontier, a mechanism that solves v positions per loop, whose speed is priced by the training contract: v ~ n_train/T_train (exponent 0.98 +/- 0.04, R^2=0.99), exactly unity under T=n training. SGD selects a frontier matching the minimum the contract demands; granting more test-time loops than ever trained rescues late positions at fixed input length, yielding a principled halting rule T* = ceil(n / v-hat). (2) Architecture prior, not expressivity, picks the algorithm: standard-depth transformers learn parallel scans on this family; weight tying flips the selection to the serial frontier, even when positional addressing for a log-depth scan is supplied. At matched depth and parameters, untied models extrapolate worst and fail to learn A5 at all. (3) The walls are not where circuit complexity says: NC1-completeness costs nothing (A5 generalizes fully), while group order does (S5's 120x120 operator deadlocks joint learning) -- and an operator-first curriculum dissolves the wall in every seed. (4) Mechanisms are portable, not mandatable: warm-starting across budget contracts transfers the algorithm in every seed, re-pricing its speed, while imposing seriality through the input schedule fails where free training succeeds. These results are invisible to standard instruments, which provably saturate at the fixed points trained loops converge to. We introduce a head instrument, the convergence-time scaling tau(n,i), validate it causally via damage cones whose slope reproduces v, and show in-distribution head measurements predict out-of-distribution fate where tail metrics do not. Results replicate on the public easy-to-hard benchmark.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Builds on 28 of its 33 references that are in the map. First posted on arXiv 22 Jul 2026.

Builds on

28 of 33 references.

References · 28

  1. 2022Locating and Editing Factual Associations in GPTcited by 2,691
  2. 2023Towards Automated Circuit Discovery for Mechanistic Interpretabilitycited by 724
  3. 2018Universal Transformerscited by 647
  4. 2021Knowledge Neurons in Pretrained Transformerscited by 601
  5. 2025Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approachcited by 415
  6. 2023Function Vectors in Large Language Modelscited by 308
  7. 2022Transformers Learn Shortcuts to Automatacited by 268
  8. 2023Does Localization Inform Editing? Surprising Differences in Causality-Based Localization vs. Knowledge Editing in Language Modelscited by 243
  9. 2022The Parallelism Tradeoff: Limitations of Log-Precision Transformerscited by 193
  10. 2024The Illusion of State in State-Space Modelscited by 187
  11. 2023Looped Transformers are Better at Learning Learning Algorithmscited by 149
  12. 2024Language-Specific Neurons: The Key to Multilingual Capabilities in Large Language Modelscited by 126
16 more
  1. 2021Can You Learn an Algorithm? Generalizing from Easy to Hard Problems with Recurrent Networkscited by 117
  2. 2022The CLRS Algorithmic Reasoning Benchmarkcited by 106
  3. 2024Looped Transformers for Length Generalizationcited by 99
  4. 2024Why are Sensitive Functions Hard for Transformers?cited by 74
  5. 2022Finding Skill Neurons in Pre-trained Transformer-based Language Modelscited by 66
  6. 2024Can Looped Transformers Learn to Implement Multi-step Gradient Descent for In-context Learning?cited by 65
  7. 2024Knowledge Circuits in Pretrained Transformerscited by 60
  8. 2025A Little Depth Goes a Long Way: The Expressive Power of Log-Depth Transformerscited by 53
  9. 2022Path Independent Equilibrium Models Can Better Exploit Test-Time Computationcited by 38
  10. 2026Loop, Think, & Generalize: Implicit Reasoning in Recurrent-Depth Transformerscited by 37
  11. 2026A Mechanistic Analysis of Looped Reasoning Language Modelscited by 33
  12. 2025Latent Chain-of-Thought? Decoding the Depth-Recurrent Transformercited by 27
  13. 2025Enhancing Auto-regressive Chain-of-Thought through Loop-Aligned Reasoningcited by 19
  14. 2023Neural Algorithmic Reasoning Without Intermediate Supervisioncited by 17
  15. 2026Step-resolved data attribution for looped transformerscited by 5
  16. 2026DifFRACT: Diffusion Feature Reconstruction and Attribution for Circuit Tracingcited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 6.

6 papers.