mlsanityData to 14 Oct 2026

Cognitive Behaviors that Enable Self-Improving Reasoners, or, Four Habits of Highly Effective STaRs

arXiv 2503.01307 ↗First posted 3 Mar 2025cs.CL cs.LGcited by 377

On the map: LLM reasoning → Reinforcement learning for reasoning

also in 1 more
Abstract (arXiv)

Test-time inference has emerged as a powerful paradigm for enabling language models to ``think'' longer and more carefully about complex challenges, much like skilled human experts. While reinforcement learning (RL) can drive self-improvement in language models on verifiable tasks, some models exhibit substantial gains while others quickly plateau. For instance, we find that Qwen-2.5-3B far exceeds Llama-3.2-3B under identical RL training for the game of Countdown. This discrepancy raises a critical question: what intrinsic properties enable effective self-improvement? We introduce a framework to investigate this question by analyzing four key cognitive behaviors -- verification, backtracking, subgoal setting, and backward chaining -- that both expert human problem solvers and successful language models employ. Our study reveals that Qwen naturally exhibits these reasoning behaviors, whereas Llama initially lacks them. In systematic experimentation with controlled behavioral datasets, we find that priming Llama with examples containing these reasoning behaviors enables substantial improvements during RL, matching or exceeding Qwen's performance. Importantly, the presence of reasoning behaviors, rather than correctness of answers, proves to be the critical factor -- models primed with incorrect solutions containing proper reasoning patterns achieve comparable performance to those trained on correct solutions. Finally, leveraging continued pretraining with OpenWebMath data, filtered to amplify reasoning behaviors, enables the Llama model to match Qwen's self-improvement trajectory. Our findings establish a fundamental relationship between initial reasoning behaviors and the capacity for improvement, explaining why some language models effectively utilize additional computation while others plateau.

Key facts. Cited by 377 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 31 of its 45 references that are in the map. First posted on arXiv 3 Mar 2025.

Builds on

31 of 45 references.

Closest ancestors · 7

Same direction, newest first

  1. 2025On the Emergence of Thinking in LLMs I: Searching for the Right Intuition10 Feb
  2. 2025Process Reinforcement through Implicit Rewards3 Feb
  3. 2024Improve Mathematical Reasoning in Language Models by Automated Process Supervision5 Jun
  4. 2024Teaching Large Language Models to Reason with Reinforcement Learning7 Mar
  5. 2023Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotations14 Dec
  6. 2023Let's Verify Step by Step31 May
  7. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 24

  1. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  2. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  3. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,211
  4. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,879
  5. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 2,057
  6. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,986
  7. 2024Large Language Monkeys: Scaling Inference Compute with Repeated Samplingcited by 991
  8. 2022STaR: Bootstrapping Reasoning With Reasoningcited by 940
  9. 2024Training Language Models to Self-Correct via Reinforcement Learningcited by 426
  10. 2024Quiet-STaR: Language Models Can Teach Themselves to Think Before Speakingcited by 301
  11. 2024Recursive Introspection: Teaching Language Model Agents How to Self-Improvecited by 165
  12. 2024Stream of Search (SoS): Learning to Search in Languagecited by 145
12 more
  1. 2025LLMs Can Easily Learn to Reason from Demonstrations Structure, not content, is what matters!cited by 132
  2. 2024Common 7B Language Models Already Possess Strong Math Capabilitiescited by 109
  3. 2025Towards System 2 Reasoning in LLMs: Learning How to Think With Meta Chain-of-Thoughtcited by 109
  4. 2021Abstraction and Analogy-Making in Artificial Intelligencecited by 103
  5. 2024Beyond A*: Better Planning with Transformers via Search Dynamics Bootstrappingcited by 90
  6. 2023Strategic Reasoning with Language Modelscited by 64
  7. 2021Scaling Scaling Laws with Board Gamescited by 63
  8. 2024A Comparative Study on Reasoning Patterns of OpenAI's o1 Modelcited by 63
  9. 2024Physics of Language Models: Part 2.2, How to Learn From Mistakes on Grade-School Math Problemscited by 50
  10. 2023Training Chain-of-Thought via Latent-Variable Inferencecited by 47
  11. 2024Self-Explore: Enhancing Mathematical Reasoning in Language Models with Fine-grained Rewardscited by 31
  12. 2024Mastering Board Games by External and Internal Planning with Language Modelscited by 31

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 256; 2026: 121.

377 papers.

Sort:

By area:

  1. 2025Emerging Properties in Unified Multimodal Pretrainingcited by 897
  2. 2025Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoningcited by 559
  3. 2025SimpleRL-Zoo: Investigating and Taming Zero Reinforcement Learning for Open Base Models in the Wildcited by 511
  4. 2025Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Modelscited by 384
  5. 2025Reinforcement Learning for Reasoning in Large Language Models with One Training Examplecited by 295
  6. 2025Reasoning with Exploration: An Entropy Perspectivecited by 281
  7. 2025DeepMath-103K: A Large-Scale, Challenging, Decontaminated, and Verifiable Mathematical Dataset for Advancing Reasoningcited by 280
  8. 2025Spurious Rewards: Rethinking Training Signals in RLVRcited by 209
  9. 2025The Landscape of Agentic Reinforcement Learning for LLMs: A Surveycited by 205
  10. 2025ProRL: Prolonged Reinforcement Learning Expands Reasoning Boundaries in Large Language Modelscited by 194
  11. 2025ThinkPrune: Pruning Long Chain-of-Thought of LLMs via Reinforcement Learningcited by 192
  12. 2025A Survey of Reinforcement Learning for Large Reasoning Modelscited by 180
38 more
  1. 2025Agentic Reinforced Policy Optimizationcited by 172
  2. 2025The Unreasonable Effectiveness of Entropy Minimization in LLM Reasoningcited by 166
  3. 2025The Surprising Effectiveness of Negative Reinforcement in LLM Reasoningcited by 162
  4. 2025Echo Chamber: RL Post-training Amplifies Behaviors Learned in Pretrainingcited by 129
  5. 2025Beyond Binary Rewards: Training LMs to Reason About Their Uncertaintycited by 117
  6. 2025OpenCodeReasoning: Advancing Data Distillation for Competitive Codingcited by 113
  7. 2025OctoThinker: Mid-training Incentivizes Reinforcement Learning Scalingcited by 113
  8. 2025Self-Evolving Curriculum for LLM Reasoningcited by 112
  9. 2025DeepSeek-R1 Thoughtology: Let's think about LLM Reasoningcited by 92
  10. 2025Curriculum Reinforcement Learning from Easy to Hard Tasks Improves LLM Reasoningcited by 92
  11. 2025Think Silently, Think Fast: Dynamic Latent Compression of LLM Reasoning Chainscited by 85
  12. 2025Grounded Reinforcement Learning for Visual Reasoningcited by 83
  13. 2025ReMA: Learning to Meta-think for LLMs with Multi-Agent Reinforcement Learningcited by 79
  14. 2025JudgeLRM: Large Reasoning Models as a Judgecited by 78
  15. 2025On the Interplay of Pre-Training, Mid-Training, and RL on Reasoning Language Modelscited by 69
  16. 2025Nemotron-Research-Tool-N1: Exploring Tool-Using Language Models with Reinforced Reasoningcited by 68
  17. 2025Can Large Reasoning Models Self-Train?cited by 68
  18. 2026Self-Distillation Zero: Self-Revision Turns Binary Rewards into Dense Supervisioncited by 68
  19. 2025Reasoning Models Better Express Their Confidencecited by 65
  20. 2025Blending Supervised and Reinforcement Fine-Tuning with Prefix Samplingcited by 60
  21. 2025One Token to Fool LLM-as-a-Judgecited by 60
  22. 2025UFT: Unifying Supervised and Reinforcement Fine-Tuningcited by 59
  23. 2025e3: Learning to Explore Enables Extrapolation of Test-Time Compute for LLMscited by 59
  24. 2025SRPO: Enhancing Multimodal LLM Reasoning via Reflection-Aware Reinforcement Learningcited by 57
  25. 2025Training a Scientific Reasoning Model for Chemistrycited by 56
  26. 2026Endless Terminals: Scaling RL Environments for Terminal Agentscited by 54
  27. 2025A Survey on Large Language Models for Mathematical Reasoningcited by 52
  28. 2025Stop Summation: Min-Form Credit Assignment Is All Process Reward Model Needs for Reasoningcited by 50
  29. 2025SynLogic: Synthesizing Verifiable Reasoning Data at Scale for Learning Logical Reasoning and Beyondcited by 50
  30. 2025RLVE: Scaling Up Reinforcement Learning for Language Models with Adaptive Verifiable Environmentscited by 50
  31. 2025MT-R1-Zero: Advancing LLM-based Machine Translation via R1-Zero-like Reinforcement Learningcited by 48
  32. 2025Scaling Laws of Synthetic Data for Language Modelscited by 46
  33. 2025Crosslingual Reasoning through Test-Time Scalingcited by 46
  34. 2025An Empirical Study on Reinforcement Learning for Reasoning-Search Interleaved LLM Agentscited by 46
  35. 2025Consistent Paths Lead to Truth: Self-Rewarding Reinforcement Learning for LLM Reasoningcited by 46
  36. 2025Rethinking Reflection in Pre-Trainingcited by 43
  37. 2025Depth-Breadth Synergy in RLVR: Unlocking LLM Reasoning Gains with Adaptive Explorationcited by 43
  38. 2025From f(x) and g(x) to f(g(x)): LLMs Learn New Skills in RL by Composing Old Onescited by 42

50 most cited of 377 All 377 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024RL-STaR: Theoretical Analysis of Reinforcement Learning Frameworks for Self-Taught Reasoner
  2. 2025Self-rewarding correction for mathematical reasoning
  3. 2025Reward Is Enough: LLMs Are In-Context Reinforcement Learners
  4. 2026Reasoning Promotes Robustness in Theory of Mind Tasks
  5. 2026Test-time Recursive Thinking: Self-Improvement without External Feedback
  6. 2026Latent Reward Steering: An Adaptive Inference-Time Framework that Implicitly Promotes Cognitive Behaviors in Reasoning LLMs
  7. 2025Towards Understanding Self-play for LLM Reasoning
  8. 2025Incentivizing LLMs to Self-Verify Their Answers