mlsanityData to 7 Oct 2026

Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach

arXiv 2502.05171 ↗First posted 7 Feb 2025cs.LG cs.CLcited by 372

On the map: LLM reasoning → Test-time compute and efficient thinking

Abstract (arXiv)

We study a novel language model architecture that is capable of scaling test-time computation by implicitly reasoning in latent space. Our model works by iterating a recurrent block, thereby unrolling to arbitrary depth at test-time. This stands in contrast to mainstream reasoning models that scale up compute by producing more tokens. Unlike approaches based on chain-of-thought, our approach does not require any specialized training data, can work with small context windows, and can capture types of reasoning that are not easily represented in words. We scale a proof-of-concept model to 3.5 billion parameters and 800 billion tokens. We show that the resulting model can improve its performance on reasoning benchmarks, sometimes dramatically, up to a computation load equivalent to 50 billion parameters.

Key facts. Cited by 372 papers in the map (data through 7 Oct 2026). Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning. Builds on 84 of its 187 references that are in the map. First posted on arXiv 7 Feb 2025.

Builds on

84 of 187 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024Deliberation in Latent Space via Differentiable Cache Augmentation23 Dec
  2. 2024Compressed Chain of Thought: Efficient Reasoning Through Dense Representations17 Dec

Other references · 82

  1. 2017Attention Is All You Needcited by 59,932
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  5. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  6. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  7. 2020Scaling Laws for Neural Language Modelscited by 6,411
  8. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,146
  9. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  10. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  11. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  12. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,067
38 more
  1. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  2. 2021Program Synthesis with Large Language Modelscited by 3,509
  3. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  4. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  5. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  6. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  7. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,928
  8. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  9. 2020GLU Variants Improve Transformercited by 1,724
  10. 2017Decoupled Weight Decay Regularizationcited by 1,722
  11. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,479
  12. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,359
  13. 2024Training Large Language Models to Reason in a Continuous Latent Spacecited by 763
  14. 2024DeepSeek-V3 Technical Reportcited by 724
  15. 2019Compressive Transformers for Long-Range Sequence Modellingcited by 699
  16. 2019MathQA: Towards Interpretable Math Word Problem Solving with Operation-Based Formalismscited by 653
  17. 2014Neural Turing Machinescited by 630
  18. 2018Universal Transformerscited by 613
  19. 2019Deep Equilibrium Modelscited by 603
  20. 2024OLMo: Accelerating the Science of Language Modelscited by 599
  21. 2024BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructionscited by 527
  22. 2024StarCoder 2 and The Stack v2: The Next Generationcited by 484
  23. 2016Adaptive Computation Time for Recurrent Neural Networkscited by 479
  24. 2023Ring Attention with Blockwise Transformers for Near-Infinite Contextcited by 448
  25. 2019Reducing Transformer Depth on Demand with Structured Dropoutcited by 434
  26. 2024Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothingcited by 341
  27. 2024LayerSkip: Enabling Early Exit Inference and Self-Speculative Decodingcited by 302
  28. 20242 OLMo 2 Furiouscited by 302
  29. 2024Quiet-STaR: Language Models Can Teach Themselves to Think Before Speakingcited by 292
  30. 2024Step-DPO: Step-wise Preference Optimization for Long-chain Reasoning of LLMscited by 265
  31. 2024HelpSteer2: Open-source dataset for training top-performing reward modelscited by 232
  32. 2024From Explicit CoT to Implicit CoT: Learning to Internalize CoT Step by Stepcited by 225
  33. 2024MobileLLM: Optimizing Sub-billion Parameter Language Models for On-Device Use Casescited by 216
  34. 2024OpenMathInstruct-2: Accelerating AI for Math with Massive Open-Source Instruction Datacited by 194
  35. 2021BookSum: A Collection of Datasets for Long-form Narrative Summarizationcited by 184
  36. 2023OctoPack: Instruction Tuning Code Large Language Modelscited by 178
  37. 2024Lessons from the Trenches on Reproducible Evaluation of Language Modelscited by 172
  38. 2024LongWriter: Unleashing 10,000+ Word Generation from Long Context LLMscited by 142

50 most cited of 82 All 82 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 131; 2026: 241.

372 papers.

Sort:

By area:

  1. 2025Dream 7B: Diffusion Large Language Modelscited by 515
  2. 2025Stop Overthinking: A Survey on Efficient Reasoning for Large Language Modelscited by 456
  3. 2025Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Modelscited by 381
  4. 2025Chain of Thought Monitorability: A New and Fragile Opportunity for AI Safetycited by 222
  5. 2025CODI: Compressing Chain-of-Thought into Continuous Space via Self-Distillationcited by 216
  6. 2025A Survey on Test-Time Scaling in Large Language Models: What, How, Where, and How Well?cited by 215
  7. 2025ThinkPrune: Pruning Long Chain-of-Thought of LLMs via Reinforcement Learningcited by 187
  8. 2025When More is Less: Understanding Chain-of-Thought Length in LLMscited by 182
  9. 2025A Survey of Reinforcement Learning for Large Reasoning Modelscited by 175
  10. 2025Hierarchical Reasoning Modelcited by 167
  11. 2025Scaling Latent Reasoning via Looped Language Modelscited by 145
  12. 2025LLM Post-Training: A Deep Dive into Reasoning Large Language Modelscited by 129
38 more
  1. 2025OpenVLThinker: Complex Vision-Language Reasoning via Iterative SFT-RL Cyclescited by 122
  2. 2025Mixture-of-Recursions: Learning Dynamic Recursive Depths for Adaptive Token-Level Computationcited by 120
  3. 2025When Chain of Thought is Necessary, Language Models Struggle to Evade Monitorscited by 81
  4. 2025Monet: Reasoning in Latent Visual Space Beyond Images and Languagecited by 75
  5. 2025A Survey on Latent Reasoningcited by 71
  6. 2025DeltaProduct: Improving State-Tracking in Linear RNNs via Householder Productscited by 69
  7. 2025SealQA: Raising the Bar for Reasoning in Search-Augmented Language Modelscited by 65
  8. 2025Dynamic Chunking for End-to-End Hierarchical Sequence Modelingcited by 65
  9. 2025SIM-CoT: Supervised Implicit Chain-of-Thoughtcited by 63
  10. 2025Inner Thinking Transformer: Leveraging Dynamic Depth Scaling to Foster Adaptive Internal Thinkingcited by 56
  11. 2025InftyThink: Breaking the Length Limits of Long-Context Reasoning in Large Language Modelscited by 53
  12. 2025Think Before Recommend: Unleashing the Latent Reasoning Power for Sequential Recommendationcited by 50
  13. 2025A Little Depth Goes a Long Way: The Expressive Power of Log-Depth Transformerscited by 49
  14. 2025Inverse Scaling in Test-Time Computecited by 49
  15. 2025Do Language Models Use Their Depth Efficiently?cited by 48
  16. 2026LoopFormer: Elastic-Depth Looped Transformers for Latent Reasoning via Shortcut Modulationcited by 48
  17. 2025Reinforced Latent Reasoning for LLM-based Recommendationcited by 46
  18. 2026Parcae: Scaling Laws For Stable Looped Language Modelscited by 45
  19. 2025PonderLM: Pretraining Language Models to Ponder in Continuous Spacecited by 44
  20. 2025Reasoning to Learn from Latent Thoughtscited by 43
  21. 2025Implicit Reasoning in Large Language Models: A Comprehensive Surveycited by 42
  22. 2025Feedback Descent: Open-Ended Text Optimization via Pairwise Comparisoncited by 42
  23. 2025Latent Sketchpad: Sketching Visual Thoughts to Elicit Multimodal Reasoning in MLLMscited by 41
  24. 2025Teaching Pretrained Language Models to Think Deeper with Retrofitted Recurrencecited by 41
  25. 2025Answer Convergence as a Signal for Early Stopping in Reasoningcited by 38
  26. 2025ParaThinker: Native Parallel Thinking as a New Paradigm to Scale LLM Test-time Computecited by 36
  27. 2025Harnessing the Reasoning Economy: A Survey of Efficient Reasoning for Large Language Modelscited by 35
  28. 2025Same Task, Different Circuits: Disentangling Modality-Specific Mechanisms in VLMscited by 35
  29. 2025LLMs are Single-threaded Reasoners: Demystifying the Working Mechanism of Soft Thinkingcited by 34
  30. 2025Continuous Chain of Thought Enables Parallel Exploration and Reasoningcited by 32
  31. 2025Soft Tokens, Hard Truthscited by 31
  32. 2025Enabling Agents to Communicate Entirely in Latent Spacecited by 31
  33. 2025Hybrid Latent Reasoning via Reinforcement Learningcited by 30
  34. 2025Multiverse: Your Language Models Secretly Decide How to Parallelize and Merge Generationcited by 30
  35. 2025Energy-Based Transformers are Scalable Learners and Thinkerscited by 30
  36. 2025Parallel Scaling Law for Language Modelscited by 29
  37. 2025Topology of Reasoning: Understanding Large Reasoning Models through Reasoning Graph Propertiescited by 28
  38. 2025BudgetThinker: Empowering Budget-aware LLM Reasoning with Control Tokenscited by 27

50 most cited of 372 All 372 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025The Art of Scaling Test-Time Compute for Large Language Models
  2. 2026Thinking Deeper, Not Longer: Memory-Efficient Test-Time Reasoning with Depth-Recurrent Transformers for Compositional Generalization
  3. 2025Reasoning on a Budget: A Survey of Adaptive and Controllable Test-Time Compute in LLMs
  4. 2026ThinkBooster: A Unified Framework for Seamless Test-Time Scaling of LLM Reasoning
  5. 2025Seek in the Dark: Reasoning via Test-Time Instance-Level Policy Gradient in Latent Space
  6. 2025ReProbe: Efficient Test-Time Scaling of Multi-Step Reasoning by Probing Internal States of Large Language Models
  7. 2026Unlocking the Working Memory of Large Language Models for Latent Reasoning
  8. 2026LaTER: Efficient Test-Time Reasoning via Latent Exploration and Explicit Verification