mlsanityData to 7 Oct 2026

Reasoning with Latent Thoughts: On the Power of Looped Transformers

arXiv 2502.17416 ↗First posted 24 Feb 2025cs.CL cs.AI cs.LGcited by 223

On the map: LLM reasoning → Test-time compute and efficient thinking

Abstract (arXiv)

Large language models have shown remarkable reasoning abilities and scaling laws suggest that large parameter count, especially along the depth axis, is the primary driver. In this work, we make a stronger claim -- many reasoning problems require a large depth but not necessarily many parameters. This unlocks a novel application of looped models for reasoning. Firstly, we show that for many synthetic reasoning problems like addition, p-hop induction, and math problems, a k-layer transformer looped L times nearly matches the performance of a kL-layer non-looped model, and is significantly better than a k-layer model. This is further corroborated by theoretical results showing that many such reasoning problems can be solved via iterative algorithms, and thus, can be solved effectively using looped models with nearly optimal depth. Perhaps surprisingly, these benefits also translate to practical settings of language modeling -- on many downstream reasoning tasks, a language model with k-layers looped L times can be competitive to, if not better than, a kL-layer language model. In fact, our empirical analysis reveals an intriguing phenomenon: looped and non-looped models exhibit scaling behavior that depends on their effective depth, akin to the inference-time scaling of chain-of-thought (CoT) reasoning. We further elucidate the connection to CoT reasoning by proving that looped models implicitly generate latent thoughts and can simulate T steps of CoT with T loops. Inspired by these findings, we also present an interesting dichotomy between reasoning and memorization, and design a looping-based regularization that is effective on both fronts.

Key facts. Cited by 223 papers in the map (data through 7 Oct 2026). Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning. Builds on 45 of its 65 references that are in the map. First posted on arXiv 24 Feb 2025.

Builds on

45 of 65 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024On the Inductive Bias of Stacking Towards Improving Reasoning27 Sep

Other references · 44

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  3. 2020Scaling Laws for Neural Language Modelscited by 6,411
  4. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,146
  5. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  6. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  7. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  8. 2022Emergent Abilities of Large Language Modelscited by 2,483
  9. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  10. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,538
  11. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 896
  12. 2019DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphscited by 883
32 more
  1. 2021Show Your Work: Scratchpads for Intermediate Computation with Language Modelscited by 871
  2. 2021Are NLP Models really able to Solve Simple Math Word Problems?cited by 861
  3. 2018CoQA: A Conversational Question Answering Challengecited by 674
  4. 2017The Implicit Bias of Gradient Descent on Separable Datacited by 649
  5. 2018Universal Transformerscited by 613
  6. 2019Deep Equilibrium Modelscited by 603
  7. 2020TyDi QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse Languagescited by 486
  8. 2018The Web as a Knowledge-base for Answering Complex Questionscited by 409
  9. 2018QuAC : Question Answering in Contextcited by 379
  10. 2021A Diverse Corpus for Evaluating and Developing English Math Word Problem Solverscited by 314
  11. 2023Towards Revealing the Mystery behind Chain of Thought: A Theoretical Perspectivecited by 293
  12. 2022UL2: Unifying Language Learning Paradigmscited by 292
  13. 2022Transformers Learn Shortcuts to Automatacited by 260
  14. 2024Chain of Thought Empowers Transformers to Solve Inherently Serial Problemscited by 256
  15. 2023Looped Transformers as Programmable Computerscited by 224
  16. 2024MobileLLM: Optimizing Sub-billion Parameter Language Models for On-Device Use Casescited by 216
  17. 2022The Parallelism Tradeoff: Limitations of Log-Precision Transformerscited by 191
  18. 2021Inductive Biases and Variable Creation in Self-Attention Mechanismscited by 144
  19. 2024Physics of Language Models: Part 3.3, Knowledge Capacity Scaling Lawscited by 137
  20. 2021Investigating the Limitations of Transformers with Simple Arithmetic Taskscited by 136
  21. 2023Looped Transformers are Better at Learning Learning Algorithmscited by 135
  22. 2023Teaching Arithmetic to Small Transformerscited by 121
  23. 2021Can You Learn an Algorithm? Generalizing from Easy to Hard Problems with Recurrent Networkscited by 108
  24. 2024Physics of Language Models: Part 2.1, Grade-School Math and the Hidden Reasoning Processcited by 107
  25. 2022Understanding Contrastive Learning Requires Incorporating Inductive Biasescited by 96
  26. 2022Same Pre-training Loss, Better Downstream: Implicit Bias Matters for Language Modelscited by 84
  27. 2024Transformers, parallel computation, and logarithmic depthcited by 75
  28. 2024Understanding Transformer Reasoning Capabilities via Graph Algorithmscited by 64
  29. 2024Can Looped Transformers Learn to Implement Multi-step Gradient Descent for In-context Learning?cited by 60
  30. 2024Simulation of Graph Algorithms with Looped Transformerscited by 29
  31. 2024What Can Transformer Learn with Varying Depth? Case Studies on Sequence Learning Taskscited by 19
  32. 2024On the Role of Depth and Looping for In-Context Learning with Task Diversitycited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 63; 2026: 160.

223 papers.

Sort:

By area:

  1. 2025Stop Overthinking: A Survey on Efficient Reasoning for Large Language Modelscited by 456
  2. 2025Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Modelscited by 381
  3. 2025Dynamic Early Exit in Reasoning Modelscited by 217
  4. 2025CODI: Compressing Chain-of-Thought into Continuous Space via Self-Distillationcited by 216
  5. 2025A Survey on Test-Time Scaling in Large Language Models: What, How, Where, and How Well?cited by 215
  6. 2025Reasoning Models Can Be Effective Without Thinkingcited by 182
  7. 2025Scaling Latent Reasoning via Looped Language Modelscited by 145
  8. 2025Mixture-of-Recursions: Learning Dynamic Recursive Depths for Adaptive Token-Level Computationcited by 120
  9. 2025Thinkless: LLM Learns When to Thinkcited by 107
  10. 2025Efficient Reasoning Models: A Surveycited by 93
  11. 2025Think Silently, Think Fast: Dynamic Latent Compression of LLM Reasoning Chainscited by 81
  12. 2025Reasoning Beyond Language: A Comprehensive Survey on Latent Chain-of-Thought Reasoningcited by 72
38 more
  1. 2025A Survey on Latent Reasoningcited by 71
  2. 2025SIM-CoT: Supervised Implicit Chain-of-Thoughtcited by 63
  3. 2025BREAD: Branched Rollouts from Expert Anchors Bridge SFT & RL for Reasoningcited by 57
  4. 2025Cache-to-Cache: Direct Semantic Communication Between Large Language Modelscited by 54
  5. 2025A Little Depth Goes a Long Way: The Expressive Power of Log-Depth Transformerscited by 49
  6. 2026LoopFormer: Elastic-Depth Looped Transformers for Latent Reasoning via Shortcut Modulationcited by 48
  7. 2026Parcae: Scaling Laws For Stable Looped Language Modelscited by 45
  8. 2025PonderLM: Pretraining Language Models to Ponder in Continuous Spacecited by 44
  9. 2025Implicit Reasoning in Large Language Models: A Comprehensive Surveycited by 42
  10. 2025Teaching Pretrained Language Models to Think Deeper with Retrofitted Recurrencecited by 41
  11. 2025Scalable Chain of Thoughts via Elastic Reasoningcited by 32
  12. 2025Soft Tokens, Hard Truthscited by 31
  13. 2025Energy-Based Transformers are Scalable Learners and Thinkerscited by 30
  14. 2025Coevolutionary Continuous Discrete Diffusion: Make Your Diffusion Language Model a Latent Reasonercited by 28
  15. 2026A Mechanistic Analysis of Looped Reasoning Language Modelscited by 27
  16. 2025LaDiR: Latent Diffusion Enhances LLMs for Text Reasoningcited by 25
  17. 2025Encode, Think, Decode: Scaling test-time reasoning with recursive latent thoughtscited by 23
  18. 2025Parallel Loop Transformer for Efficient Test-Time Computation Scalingcited by 23
  19. 2025Parallel Continuous Chain-of-Thought with Jacobi Iterationcited by 21
  20. 2026Fixed-Point Reasoners: Stable and Adaptive Deep Looped Transformerscited by 21
  21. 2026How Much Is One Recurrence Worth? Iso-Depth Scaling Laws for Looped Language Modelscited by 20
  22. 2025Towards Concise and Adaptive Thinking in Large Reasoning Models: A Surveycited by 18
  23. 2025OThink-R1: Intrinsic Fast/Slow Thinking Mode Switching for Over-Reasoning Mitigationcited by 17
  24. 2025ViT3: Unlocking Test-Time Training in Visioncited by 15
  25. 2025A Formal Comparison Between Chain of Thought and Latent Thoughtcited by 14
  26. 2026LoopViT: Scaling Visual ARC with Looped Transformerscited by 14
  27. 2025Adaptive GoGI-Skip: Coupling Goal-Gradient Importance with Dynamic Uncertainty for Efficient Reasoningcited by 12
  28. 2025SynAdapt: Learning Adaptive Reasoning in Large Language Models via Synthetic Continuous Chain-of-Thoughtcited by 12
  29. 2025Think-at-Hard: Dynamic Looped Transformers for Improved Reasoningcited by 12
  30. 2026Equilibrium Reasoners: Learning Attractors Enables Scalable Reasoningcited by 12
  31. 2026Training-Free Looped Transformerscited by 12
  32. 2026SMELT: Scaling Laws for Compute-Matched MoE Looped Transformerscited by 12
  33. 2026Depth-Recurrent Attention Mixtures: Giving Latent Reasoning the Attention it Deservescited by 11
  34. 2026Adaptive Loops and Memory in Transformers: Think Harder or Know More?cited by 11
  35. 2026ELT: Elastic Looped Transformers for Visual Generationcited by 11
  36. 2026Hyperloop Transformerscited by 11
  37. 2026Memory-Efficient Looped Transformer: Decoupling Compute from Memory in Looped Language Modelscited by 11
  38. 2026Solve the Loop: Attractor Models for Language and Reasoningcited by 11

50 most cited of 223 All 223 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Bridging the Gap Between Latent and Explicit Reasoning with Looped Transformers
  2. 2025Advancing Reasoning in Large Language Models: Promising Methods and Approaches
  3. 2026Closing the Loop: Practical Training Recipes for Looped Language Models
  4. 2026LoopCD: Loop-wise Contrastive Decoding for Improving Reasoning in Looped Language Models
  5. 2022Language Models Are Greedy Reasoners: A Systematic Formal Analysis of Chain-of-Thought
  6. 2025Finding the Minimal Parameter Budget for Implicit Reasoning: A Data Complexity Driven Scaling Law for Language Models
  7. 2025Reasoning Models Reason Well, Until They Don't
  8. 2025Hierarchical Reasoning Models: Perspectives and Misconceptions