mlsanityData to 7 Oct 2026

Teaching Pretrained Language Models to Think Deeper with Retrofitted Recurrence

arXiv 2511.07384 ↗First posted 10 Nov 2025cs.CL cs.AI cs.LGcited by 41

On the map: LLM reasoning → Test-time compute and efficient thinking

Abstract (arXiv)

Recent advances in depth-recurrent language models show that recurrence can decouple train-time compute and parameter count from test-time compute. In this work, we study how to convert existing pretrained non-recurrent language models into depth-recurrent models. We find that using a curriculum of recurrences to increase the effective depth of the model over the course of training preserves performance while reducing total computational cost. In our experiments, on mathematics, we observe that converting pretrained models to recurrent ones results in better performance at a given compute budget than simply post-training the original non-recurrent language model.

Key facts. Cited by 41 papers in the map (data through 7 Oct 2026). Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning. Builds on 54 of its 96 references that are in the map. First posted on arXiv 10 Nov 2025.

Builds on

54 of 96 references.

Closest ancestors · 10

Same direction, newest first

  1. 2025Encode, Think, Decode: Scaling test-time reasoning with recursive latent thoughts8 Oct
  2. 2025NVIDIA Nemotron Nano 2: An Accurate and Efficient Hybrid Mamba-Transformer Reasoning Model20 Aug
  3. 2025A Survey on Latent Reasoning8 Jul
  4. 2025Energy-Based Transformers are Scalable Learners and Thinkers2 Jul
  5. 2025Hierarchical Reasoning Model26 Jun
  6. 2025Reasoning with Latent Thoughts: On the Power of Looped Transformers24 Feb
  7. 2025Inner Thinking Transformer: Leveraging Dynamic Depth Scaling to Foster Adaptive Internal Thinking19 Feb
  8. 2025Zero Token-Driven Deep Thinking in LLMs: Unlocking the Full Potential of Existing Parameters via Cyclic Refinement17 Feb
  9. 2025Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach7 Feb
  10. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters6 Aug

Other references · 44

  1. 2024The Llama 3 Herd of Modelscited by 14,192
  2. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  3. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  4. 2020Scaling Laws for Neural Language Modelscited by 6,411
  5. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,146
  6. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  7. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  8. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  9. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  10. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  11. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
  12. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,311
32 more
  1. 2024Training Large Language Models to Reason in a Continuous Latent Spacecited by 763
  2. 2018Universal Transformerscited by 613
  3. 2016Adaptive Computation Time for Recurrent Neural Networkscited by 479
  4. 2024ShortGPT: Layers in Large Language Models are More Redundant Than You Expectcited by 369
  5. 20242 OLMo 2 Furiouscited by 302
  6. 2024Mixture-of-Depths: Dynamically allocating compute in transformer-based language modelscited by 242
  7. 2019Depth-Adaptive Transformercited by 227
  8. 2023Looped Transformers are Better at Learning Learning Algorithmscited by 135
  9. 2024LaCo: Large Language Model Pruning via Layer Collapsecited by 124
  10. 2025Mixture-of-Recursions: Learning Dynamic Recursive Depths for Adaptive Token-Level Computationcited by 120
  11. 2021Finetuning Pretrained Transformers into RNNscited by 94
  12. 2024Looped Transformers for Length Generalizationcited by 91
  13. 2021Lessons on Parameter Sharing across Layers in Transformerscited by 85
  14. 2024Relaxed Recursive Transformers: Effective Parameter Sharing with Layer-wise LoRAcited by 72
  15. 2024Scaling Exponents Across Parameterizations and Optimizerscited by 71
  16. 2024Can Looped Transformers Learn to Implement Multi-step Gradient Descent for In-context Learning?cited by 60
  17. 2025Falcon-H1: A Family of Hybrid-Head Language Models Redefining Efficiency and Performancecited by 52
  18. 2023Spike No More: Stabilizing the Pre-training of Large Language Modelscited by 51
  19. 2024Linearizing Large Language Modelscited by 51
  20. 2025A Little Depth Goes a Long Way: The Expressive Power of Log-Depth Transformerscited by 49
  21. 2025Do Language Models Use Their Depth Efficiently?cited by 48
  22. 2025PonderLM: Pretraining Language Models to Ponder in Continuous Spacecited by 44
  23. 2020The Depth-to-Width Interplay in Self-Attentioncited by 39
  24. 2025MesaNet: Sequence Modeling by Locally Optimal Test-Time Trainingcited by 37
  25. 2025Nemotron-CC-Math: A 133 Billion-Token-Scale High Quality Math Pretraining Datasetcited by 34
  26. 2023CoTFormer: A Chain-of-Thought Driven Architecture with Budget-Adaptive Computation Cost at Inferencecited by 27
  27. 2024LoLCATs: On Low-Rank Linearizing of Large Language Modelscited by 25
  28. 2025Gemstones: A Model Suite for Multi-Faceted Scaling Lawscited by 19
  29. 2020Revisiting BFloat16 Trainingcited by 15
  30. 2024AlgoFormer: An Efficient Transformer Framework with Algorithmic Structurescited by 11
  31. 2024Loop Neural Networks for Parameter Sharingcited by 10
  32. 2025AbbIE: Autoregressive Block-Based Iterative Encoder for Efficient Sequence Modelingcited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 41.

41 papers.

Sort:

By area:

  1. 2026Parcae: Scaling Laws For Stable Looped Language Modelscited by 45
  2. 2026A Mechanistic Analysis of Looped Reasoning Language Modelscited by 27
  3. 2026How Much Is One Recurrence Worth? Iso-Depth Scaling Laws for Looped Language Modelscited by 20
  4. 2026Recurrent-Depth VLA: Implicit Test-Time Compute Scaling of Vision-Language-Action Models via Latent Iterative Reasoningcited by 17
  5. 2026Stabilizing Recurrent Dynamics for Test-Time Scalable Latent Reasoning in Looped Language Modelscited by 14
  6. 2026LoopUS: Recasting Pretrained LLMs into Looped Latent Refinement Modelscited by 13
  7. 2026Hyperloop Transformerscited by 11
  8. 2026Bridging the Gap Between Latent and Explicit Reasoning with Looped Transformerscited by 11
  9. 2026Loop as a Bridge: Can Looped Transformers Truly Link Representation Space and Natural Language Outputs?cited by 8
  10. 2026A Scalable Measure of Loss Landscape Curvature for Analyzing the Training Dynamics of LLMscited by 8
  11. 2026LoopMoE: Unifying Iterative Computation with Mixture-of-Experts for Language Modelingcited by 7
  12. 2026Understanding Dynamic Compute Allocation in Recurrent Transformerscited by 6
29 more
  1. 2026From Growing to Looping: A Unified View of Iterative Computation in LLMscited by 6
  2. 2026The Illusion of Superposition? A Principled Analysis of Latent Thinking in Language Modelscited by 6
  3. 2026Step-resolved data attribution for looped transformerscited by 5
  4. 2026Retrofitting Recurrent Depth into a Pretrained Language Model: Installation, Extrapolation, Transfer, and Retention at Two Parameter Budgetscited by 5
  5. 2026Is One Layer Enough? Understanding Inference Dynamics in Tabular Foundation Modelscited by 4
  6. 2026Loop the Loopies!cited by 4
  7. 2026The Topological Trouble With Transformerscited by 3
  8. 2026Exploration of Fast-Slow Latent Recurrence for Train-Short, Test-Long Generalizationcited by 2
  9. 2026LoopSpec: Pipelined Self-Speculative Decoding for Looped Transformerscited by 2
  10. 2026Ouroboros: Dynamic Weight Generation for Recursive Transformers via Input-Conditioned LoRA Modulationcited by 1
  11. 2026Looped Language Models Improve Compositional Tool Callingcited by 1
  12. 2026Do Language Models Need Sleep? Offline Recurrence for Improved Online Inferencecited by 0
  13. 2026Looped Diffusion Language Modelscited by 0
  14. 2026Right Direction, Wrong Step: Geometric Analysis of Finite-Step Failure in Looped Transformerscited by 0
  15. 2026Learn Your Own Thoughts: Abstract Token Curriculumcited by 0
  16. 2026Prediction Dynamics in Depth-Recurrent Language Modelscited by 0
  17. 2026WaveFront Decoding: Parallelized Self-Speculative Decoding for Looped Language Modelscited by 0
  18. 2026Attention Routing Stabilizes Early: Working-Set Inference for Recurrent Language Modelscited by 0
  19. 2026Quantizing Looped Transformers: Feedback Exposure and Calibration Blindnesscited by 0
  20. 2026Loop Dropout: Regularizing Shared Updates in Looped Language Modelscited by 0
  21. 2026Shallow Queries, Mature Values: Depth-Asynchronous Self-Speculation for Looped Transformerscited by 0
  22. 2026Improving Test-Time Scaling with Adaptive Looped Transformerscited by 0
  23. 2026How to Loop MoE: Flatten the Experts, Untie the Attentioncited by 0
  24. 2026Transformers Stop Thinking Too Early, and a Tiny LoRA Fixes Itcited by 0
  25. 2026What Makes Recurrence Effective in Looped Language Models?cited by 0
  26. 2026Procedural Core: A Compact Recurrent Initialization for Vision Transformerscited by 0
  27. 2026A Tilted Bowl Is Not a Slippery Slope: Compressing Looped Modelscited by 0
  28. 2026Scaling Laws for Looped Mixture of Expertscited by 0
  29. 2026Enabling Dynamic Computation in Looped LMscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Efficient Parallel Samplers for Recurrent-Depth Models and Their Connection to Diffusion Language Models
  2. 2026Closing the Loop: Practical Training Recipes for Looped Language Models
  3. 2024Griffin: Mixing Gated Linear Recurrences with Local Attention for Efficient Language Models
  4. 2024Autoregressive + Chain of Thought = Recurrent: Recurrence's Role in Language Models' Computability and a Revisit of Recurrent Transformer
  5. 2022Memorizing Transformers
  6. 2025Skip a Layer or Loop it? Test-Time Depth Adaptation of Pretrained LLMs
  7. 2026DreamingGoose: Staged Distillation from Autoregressive Transformers to Bidirectional Recurrent Diffusion Language Models
  8. 2026Not Too Hard, Not Too Easy: Learning from Intermediate States for LLM Structured Reasoning