mlsanityData to 7 Oct 2026

Rethinking Reflection in Pre-Training

arXiv 2504.04022 ↗First posted 5 Apr 2025cs.CL cs.AIcited by 43

On the map: LLM reasoning → Test-time compute and efficient thinking

Abstract (arXiv)

A language model's ability to reflect on its own reasoning provides a key advantage for solving complex problems. While most recent research has focused on how this ability develops during reinforcement learning, we show that it actually begins to emerge much earlier - during the model's pre-training. To study this, we introduce deliberate errors into chains-of-thought and test whether the model can still arrive at the correct answer by recognizing and correcting these mistakes. By tracking performance across different stages of pre-training, we observe that this self-correcting ability appears early and improves steadily over time. For instance, an OLMo2-7B model pre-trained on 4 trillion tokens displays self-correction on our six self-reflection tasks.

Key facts. Cited by 43 papers in the map (data through 7 Oct 2026). Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning. Builds on 27 of its 65 references that are in the map. First posted on arXiv 5 Apr 2025.

Builds on

27 of 65 references.

Closest ancestors · 4

Same direction, newest first

  1. 2025Rethinking Fine-Tuning when Scaling Test-Time Compute: Limiting Confidence Improves Mathematical Reasoning11 Feb
  2. 2025Demystifying Long Chain-of-Thought Reasoning in LLMs5 Feb
  3. 2025s1: Simple test-time scaling31 Jan
  4. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters6 Aug

Other references · 23

  1. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  2. 2020Scaling Laws for Neural Language Modelscited by 6,411
  3. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,146
  4. 2013Intriguing properties of neural networkscited by 5,114
  5. 2024Qwen2.5 Technical Reportcited by 4,713
  6. 2024GPT-4o System Cardcited by 4,467
  7. 2023Let's Verify Step by Stepcited by 3,598
  8. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
  9. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  10. 2025Understanding R1-Zero-Like Training: A Critical Perspectivecited by 1,276
  11. 2024OpenAI o1 System Cardcited by 1,220
  12. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,107
11 more
  1. 2024DeepSeek-V3 Technical Reportcited by 724
  2. 2020Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillationcited by 591
  3. 2024DataComp-LM: In search of the next generation of training sets for language modelscited by 381
  4. 2025Cognitive Behaviors that Enable Self-Improving Reasoners, or, Four Habits of Highly Effective STaRscited by 373
  5. 20242 OLMo 2 Furiouscited by 302
  6. 2023Evaluating the Logical Reasoning Ability of ChatGPT and GPT-4cited by 202
  7. 2025LLMs Can Easily Learn to Reason from Demonstrations Structure, not content, is what matters!cited by 132
  8. 2024Efficient Adversarial Training in LLMs with Continuous Attackscited by 104
  9. 2023Chain-of-Thought Hub: A Continuous Effort to Measure Large Language Models' Reasoning Performancecited by 102
  10. 2025Do Large Language Model Benchmarks Test Reliability?cited by 50
  11. 2024Procedural Knowledge in Pretraining Drives Reasoning in Large Language Modelscited by 41

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 38; 2026: 5.

43 papers.

Sort:

By area:

  1. 2025Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?cited by 948
  2. 2025Reinforcement Learning for Reasoning in Large Language Models with One Training Examplecited by 288
  3. 2025Reasoning with Exploration: An Entropy Perspectivecited by 277
  4. 2025Spurious Rewards: Rethinking Training Signals in RLVRcited by 199
  5. 2025The Landscape of Agentic Reinforcement Learning for LLMs: A Surveycited by 195
  6. 2025The Invisible Leash: Why RLVR May or May Not Escape Its Origincited by 92
  7. 2025SRPO: Enhancing Multimodal LLM Reasoning via Reflection-Aware Reinforcement Learningcited by 53
  8. 2025Advancing Multimodal Reasoning via Reinforcement Learning with Cold Startcited by 41
  9. 2025Depth-Breadth Synergy in RLVR: Unlocking LLM Reasoning Gains with Adaptive Explorationcited by 40
  10. 2025Reinforcement Learning Meets Large Language Models: A Survey of Advancements and Applications Across the LLM Lifecyclecited by 34
  11. 2025Beyond Fixed: Training-Free Variable-Length Denoising for Diffusion Large Language Modelscited by 32
  12. 2025RL Squeezes, SFT Expands: A Comparative Study of Reasoning LLMscited by 32
31 more
  1. 2025Front-Loading Reasoning: The Synergy between Pretraining and Post-Training Datacited by 26
  2. 2025GUI-Reflection: Empowering Multimodal GUI Models with Self-Reflection Behaviorcited by 22
  3. 2025Plan Then Action:High-Level Planning Guidance Reinforcement Learning for LLM Reasoningcited by 17
  4. 2025Steering LLM Reasoning Through Bias-Only Adaptationcited by 16
  5. 2025GThinker: Towards General Multimodal Reasoning via Cue-Guided Rethinkingcited by 14
  6. 2025Scalpel vs. Hammer: GRPO Amplifies Existing Capabilities, SFT Replaces Themcited by 14
  7. 2025The Debate on RLVR Reasoning Capability Boundary: Shrinkage, Expansion, or Both? A Two-Stage Dynamic Viewcited by 13
  8. 2025REST: Stress Testing Large Reasoning Models by Asking Multiple Problems at Oncecited by 12
  9. 2025Risk-Sensitive RL for Alleviating Exploration Dilemmas in Large Language Modelscited by 12
  10. 2025Position: The Hidden Costs and Measurement Gaps of Reinforcement Learning with Verifiable Rewardscited by 11
  11. 2025Efficient Reasoning Through Suppression of Self-Affirmation Reflections in Large Reasoning Modelscited by 10
  12. 2025ProxyThinker: Test-Time Guidance through Small Visual Reasonerscited by 8
  13. 2025G-KV: Decoding-Time KV Cache Eviction with Global Attentioncited by 7
  14. 2025J1: Exploring Simple Test-Time Scaling for LLM-as-a-Judgecited by 5
  15. 2025From Emergence to Control: Probing and Modulating Self-Reflection in Language Modelscited by 5
  16. 2025Prompting Large Language Models with Partial Knowledge for Answering Questions with Unseen Entitiescited by 4
  17. 2025Provable Benefit of Curriculum in Transformer Tree-Reasoning Post-Trainingcited by 4
  18. 2026Accordion-Thinking: Self-Regulated Step Summaries for Efficient and Readable LLM Reasoningcited by 4
  19. 2025The Pragmatic Mind of Machines: Tracing the Emergence of Pragmatic Competence in Large Language Modelscited by 3
  20. 2025Few-Shot Vision-Language Reasoning for Satellite Imagery via Verifiable Rewardscited by 3
  21. 2025Distributional Biases in Post-Training: A Markovian Analysis of Reasoning Trajectoriescited by 3
  22. 2025Deciphering Trajectory-Aided LLM Reasoning: An Optimization Perspectivecited by 2
  23. 2025Empowering Lightweight MLLMs with Reasoning via Long CoT SFTcited by 2
  24. 2025Expanding Reasoning Potential in Foundation Model by Learning Diverse Chains of Thought Patternscited by 2
  25. 2025Data-Centric Lessons To Improve Speech-Language Pretrainingcited by 2
  26. 2026On Advantage Estimates for Max@K Policy Gradientscited by 2
  27. 2025Does RLVR Extend Reasoning Boundaries? Investigating Capability Expansion in Vision-Language Modelscited by 1
  28. 2026OrderGrad: Optimizing Beyond the Mean with Order-Statistic Policy Gradient Estimationcited by 1
  29. 2025CAPE: Capability Achievement via Policy Executioncited by 0
  30. 2026History-Guided Iterative Visual Reasoning with Self-Correctioncited by 0
  31. 2026How Much Thinking is Enough? Quantifying and Understanding Redundancy in LLM Reasoningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Reflect, Retry, Reward: Self-Improving LLMs via Reinforcement Learning
  2. 2025Unveiling the Latent Directions of Reflection in Large Language Models
  3. 2024When Hindsight is Not 20/20: Testing Limits on Reflective Thinking in Large Language Models
  4. 2025The Mirror Loop: Recursive Non-Convergence in Generative Reasoning Systems
  5. 2025First Try Matters: Revisiting the Role of Reflection in Reasoning Models
  6. 2024Self-Reflection Makes Large Language Models Safer, Less Biased, and Ideologically Neutral
  7. 2024Self-Contrast: Better Reflection Through Inconsistent Solving Perspectives
  8. 2025ReflCtrl: Controlling LLM Reflection Efficiently via Representation Engineering