mlsanityData to 7 Oct 2026

On Expressive Power of Looped Transformers: Theoretical Analysis and Enhancement via Timestep Encoding

arXiv 2410.01405 ↗First posted 2 Oct 2024cs.LGcited by 28

Not on the map. Why?

Abstract (arXiv)

Looped Transformers provide advantages in parameter efficiency, computational capabilities, and generalization for reasoning tasks. However, their expressive power regarding function approximation remains underexplored. In this paper, we establish the approximation rate of Looped Transformers by defining the modulus of continuity for sequence-to-sequence functions. This reveals a limitation specific to the looped architecture. That is, the analysis prompts the incorporation of scaling parameters for each loop, conditioned on timestep encoding. Experiments validate the theoretical results, showing that increasing the number of loops enhances performance, with further gains achieved through the timestep encoding.

Key facts. Cited by 28 papers in the map (data through 7 Oct 2026). Builds on 35 of its 52 references that are in the map. First posted on arXiv 2 Oct 2024.

Builds on

35 of 52 references.

References · 35

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  4. 2022Scalable Diffusion Models with Transformerscited by 5,931
  5. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,155
  6. 2016Pointer Sentinel Mixture Modelscited by 2,992
  7. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  8. 2017Decoupled Weight Decay Regularizationcited by 1,722
  9. 2019Root Mean Square Layer Normalizationcited by 1,425
  10. 2016HyperNetworkscited by 1,003
  11. 2022Transformers learn in-context by gradient descentcited by 714
  12. 2022What Can Transformers Learn In-Context? A Case Study of Simple Function Classescited by 700
23 more
  1. 2022What learning algorithm is in-context learning? Investigations with linear modelscited by 671
  2. 2018Universal Transformerscited by 613
  3. 2019Deep Equilibrium Modelscited by 603
  4. 2019Are Transformers universal approximators of sequence-to-sequence functions?cited by 404
  5. 2023Towards Revealing the Mystery behind Chain of Thought: A Theoretical Perspectivecited by 293
  6. 2020ReZero is All You Need: Fast Convergence at Large Depthcited by 260
  7. 2023Looped Transformers as Programmable Computerscited by 224
  8. 2022Neural Networks and the Chomsky Hierarchycited by 214
  9. 2022The Parallelism Tradeoff: Limitations of Log-Precision Transformerscited by 191
  10. 2018Optimal approximation of continuous functions by very deep ReLU networkscited by 165
  11. 2024Stream of Search (SoS): Learning to Search in Languagecited by 141
  12. 2023Looped Transformers are Better at Learning Learning Algorithmscited by 135
  13. 2024Looped Transformers for Length Generalizationcited by 91
  14. 2021Lessons on Parameter Sharing across Layers in Transformerscited by 85
  15. 2023Are Transformers with One Layer Self-Attention Using Low-Rank Weight Matrices Universal Approximators?cited by 46
  16. 2021On the Optimal Memorization Power of ReLU Neural Networkscited by 40
  17. 2023Approximation and Estimation Ability of Transformers for Sequence-to-Sequence Functions with Infinite Dimensional Inputcited by 32
  18. 2024Simulation of Graph Algorithms with Looped Transformerscited by 29
  19. 2023Approximation Rate of the Transformer Architecture for Sequence Modelingcited by 28
  20. 2024Understanding the Expressive Power and Mechanisms of Transformer for Sequence Modelingcited by 25
  21. 2024How Well Can Transformers Emulate In-context Newton's Method?cited by 23
  22. 2023Learning to Solve Constraint Satisfaction Problems with Recurrent Transformercited by 21
  23. 2023On Enhancing Expressive Power via Compositions of Single Fixed-Size ReLU Networkcited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 6; 2026: 21.

28 papers.

Sort:

By area:

  1. 2026LoopFormer: Elastic-Depth Looped Transformers for Latent Reasoning via Shortcut Modulationcited by 48
  2. 2026Parcae: Scaling Laws For Stable Looped Language Modelscited by 45
  3. 2025Coevolutionary Continuous Discrete Diffusion: Make Your Diffusion Language Model a Latent Reasonercited by 28
  4. 2026A Mechanistic Analysis of Looped Reasoning Language Modelscited by 27
  5. 2025Enhancing Auto-regressive Chain-of-Thought through Loop-Aligned Reasoningcited by 19
  6. 2025A Formal Comparison Between Chain of Thought and Latent Thoughtcited by 14
  7. 2026Equilibrium Reasoners: Learning Attractors Enables Scalable Reasoningcited by 12
  8. 2026Training-Free Looped Transformerscited by 12
  9. 2026Hyperloop Transformerscited by 11
  10. 2026Solve the Loop: Attractor Models for Language and Reasoningcited by 11
  11. 2025To CoT or To Loop? A Formal Comparison Between Chain-of-Thought and Looped Transformerscited by 10
  12. 2026Prioritize the Process, Not Just the Outcome: Rewarding Latent Thought Trajectories Improves Reasoning in Looped Language Modelscited by 6
16 more
  1. 2024Advancing the Understanding of Fixed Point Iterations in Deep Neural Networks: A Detailed Analytical Studycited by 4
  2. 2025Approximation Bounds for Transformer Networks with Application to Regressioncited by 4
  3. 2026Loop the Loopies!cited by 4
  4. 2026Exploration of Fast-Slow Latent Recurrence for Train-Short, Test-Long Generalizationcited by 2
  5. 2026T-LoopFormer: Token-Level Elastic-Depth Looped Transformers for Latent Reasoning with Dynamic Routingcited by 1
  6. 2025ScaleNet: Scaling up Pretrained Neural Networks with Incremental Parameterscited by 0
  7. 2026Beyond What Seems Necessary: Hidden Gains from Scaling Training-Time Reasoning Length under Outcome Supervisioncited by 0
  8. 2026Exponential Sample Complexity Separation between Flat and Hierarchical Agentic Theorem Proverscited by 0
  9. 2026Loom: A Scalable Analytical Neural Computer Architecturecited by 0
  10. 2026Approximation Error Upper and Lower Bounds for H"older Class with Transformerscited by 0
  11. 2026Anti Mode-Collapse in Mean-Field Transformer via Auxiliary Variablescited by 0
  12. 2026How Many Thoughts Can a Vector Hold? The Capacity of Reasoning by Superpositioncited by 0
  13. 2026Right Direction, Wrong Step: Geometric Analysis of Finite-Step Failure in Looped Transformerscited by 0
  14. 2026Beyond the Training Horizon: Mechanisms and Limits of Length Generalization in Looped Transformerscited by 0
  15. 2026Loop Dropout: Regularizing Shared Updates in Looped Language Modelscited by 0
  16. 2026What Makes Recurrence Effective in Looped Language Models?cited by 0