mlsanityData to 14 Oct 2026

Scaling of Search and Learning: A Roadmap to Reproduce o1 from Reinforcement Learning Perspective

arXiv 2412.14135 ↗First posted 18 Dec 2024cs.AI cs.LGcited by 50

On the map: LLM reasoning → Test-time compute and efficient thinking

Abstract (arXiv)

OpenAI o1 represents a significant milestone in Artificial Inteiligence, which achieves expert-level performances on many challanging tasks that require strong reasoning ability.OpenAI has claimed that the main techinique behinds o1 is the reinforcement learining. Recent works use alternative approaches like knowledge distillation to imitate o1's reasoning style, but their effectiveness is limited by the capability ceiling of the teacher model. Therefore, this paper analyzes the roadmap to achieving o1 from the perspective of reinforcement learning, focusing on four key components: policy initialization, reward design, search, and learning. Policy initialization enables models to develop human-like reasoning behaviors, equipping them with the ability to effectively explore solution spaces for complex problems. Reward design provides dense and effective signals via reward shaping or reward modeling, which is the guidance for both search and learning. Search plays a crucial role in generating high-quality solutions during both training and testing phases, which can produce better solutions with more computation. Learning utilizes the data generated by search for improving policy, which can achieve the better performance with more parameters and more searched data. Existing open-source projects that attempt to reproduce o1 can be seem as a part or a variant of our roadmap. Collectively, these components underscore how learning and search drive o1's advancement, making meaningful contributions to the development of LLM.

Key facts. Cited by 50 papers in the map (data through 14 Oct 2026). Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning. Builds on 97 of its 229 references that are in the map. First posted on arXiv 18 Dec 2024.

Builds on

97 of 229 references.

Closest ancestors · 16

Same direction, newest first

  1. 2024Imitate, Explore, and Self-Improve: A Reproduction Report on Slow-thinking Reasoning Systems12 Dec
  2. 2024O1 Replication Journey -- Part 2: Surpassing O1-preview through Simple Distillation, Big Progress or Bitter Lesson?25 Nov
  3. 2024Marco-o1: Towards Open Reasoning Models for Open-Ended Solutions21 Nov
  4. 2024Enhancing LLM Reasoning with Reward-guided Tree Search18 Nov
  5. 2024TreeBoN: Enhancing Inference-Time Alignment with Speculative Tree-Search and Best-of-N Sampling18 Oct
  6. 2024O1 Replication Journey: A Strategic Progress Report -- Part 18 Oct
  7. 2024Training Language Models to Self-Correct via Reinforcement Learning19 Sep
  8. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters6 Aug
  9. 2024Large Language Models have Intrinsic Self-Correction Ability21 Jun
  10. 2024Toward Self-Improvement of LLMs via Imagination, Searching, and Criticizing18 Apr
  11. 2024Mirror: A Multiple-perspective Self-Reflection Method for Knowledge-rich Reasoning22 Feb
  12. 2024Learning to Check: Unleashing Potentials for Self-Correction in Large Language Models20 Feb
4 more
  1. 2024V-STaR: Training Verifiers for Self-Taught Reasoners9 Feb
  2. 2023Universal Self-Consistency for Large Language Model Generation29 Nov
  3. 2023Adaptive-Solver Framework for Dynamic Strategy Selection in Large Language Model Reasoning1 Oct
  4. 2023Automatically Correcting Large Language Models: Surveying the landscape of diverse self-correction strategies6 Aug

Other references · 81

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  4. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  5. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  6. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  7. 2020Scaling Laws for Neural Language Modelscited by 6,586
  8. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  9. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,050
  10. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  11. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,984
  12. 2024Mixtral of Expertscited by 1,823
38 more
  1. 2018World Modelscited by 1,513
  2. 2024A Survey on LLM-as-a-Judgecited by 1,423
  3. 2024Qwen2.5-Coder Technical Reportcited by 1,203
  4. 2024Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvementcited by 1,090
  5. 2024Large Language Monkeys: Scaling Inference Compute with Repeated Samplingcited by 991
  6. 2023Instruction Tuning for Large Language Models: A Surveycited by 623
  7. 2018Scalable agent alignment via reward modeling: a research directioncited by 447
  8. 2024InternLM2 Technical Reportcited by 397
  9. 2024ReST-MCTS*: LLM Self-Training via Process Reward Guided Tree Searchcited by 344
  10. 2024Rewarding Progress: Scaling Automated Process Verifiers for LLM Reasoningcited by 279
  11. 2024Navigation World Modelscited by 272
  12. 2024From r to Q^*: Your Language Model is Secretly a Q-Functioncited by 242
  13. 2024Monte Carlo Tree Search Boosts Reasoning via Iterative Preference Learningcited by 241
  14. 2022Emergent Analogical Reasoning in Large Language Modelscited by 240
  15. 2023Secrets of RLHF in Large Language Models Part I: PPOcited by 229
  16. 2024Agent Q: Advanced Reasoning and Learning for Autonomous AI Agentscited by 208
  17. 2024Meta-Rewarding Language Models: Self-Improving Alignment with LLM-as-a-Meta-Judgecited by 204
  18. 2024World Model on Million-Length Video And Language With Blockwise RingAttentioncited by 198
  19. 2024RedPajama: an Open Dataset for Training Large Language Modelscited by 182
  20. 2024Secrets of RLHF in Large Language Models Part II: Reward Modelingcited by 171
  21. 2024AlphaMath Almost Zero: Process Supervision without Processcited by 162
  22. 2024Teaching Large Language Models to Reason with Reinforcement Learningcited by 160
  23. 2024Accessing GPT-4 level Mathematical Olympiad Solutions via Monte Carlo Tree Self-refine with LLaMa-3 8Bcited by 158
  24. 2024Tree Search for Language Model Agentscited by 151
  25. 2023Progressive-Hint Prompting Improves Reasoning in Large Language Modelscited by 147
  26. 2024LongWriter: Unleashing 10,000+ Word Generation from Long Context LLMscited by 144
  27. 2024Multi-Step Reasoning with Large Language Models, a Surveycited by 143
  28. 2024Mutual Reasoning Makes Smaller LLMs Stronger Problem-Solverscited by 138
  29. 2024DPO Meets PPO: Reinforced Token Optimization for RLHFcited by 127
  30. 2024Language Models Learn to Mislead Humans via RLHFcited by 124
  31. 2024BoNBoN Alignment for Large Language Models and the Sweetness of Best-of-n Samplingcited by 121
  32. 2024Fast Best-of-N Decoding via Speculative Rejectioncited by 112
  33. 2023Let's reward step by step: Step-Level reward model as the Navigators for Reasoningcited by 102
  34. 2024OpenR: An Open Source Framework for Advanced Reasoning with Large Language Modelscited by 94
  35. 2024LLaMA-Berry: Pairwise Optimization for O1-like Olympiad-Level Mathematical Reasoningcited by 91
  36. 2024Beyond A*: Better Planning with Transformers via Search Dynamics Bootstrappingcited by 90
  37. 2024How Do Large Language Models Acquire Factual Knowledge During Pretraining?cited by 90
  38. 2024WorldDreamer: Towards General World Models for Video Generation via Predicting Masked Tokenscited by 89

50 most cited of 81 All 81 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 45; 2026: 4.

50 papers.

Sort:

By area:

  1. 2025Search-o1: Agentic Search-Enhanced Large Reasoning Modelscited by 433
  2. 2025Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Modelscited by 384
  3. 2025From System 1 to System 2: A Survey of Reasoning Large Language Modelscited by 320
  4. 2024HuatuoGPT-o1, Towards Medical Complex Reasoning with LLMscited by 265
  5. 2025DAST: Difficulty-Adaptive Slow-Thinking for Large Reasoning Modelscited by 158
  6. 2025LLM Post-Training: A Deep Dive into Reasoning Large Language Modelscited by 135
  7. 2025A Survey of Efficient Reasoning for Large Reasoning Models: Language, Multimodality, and Beyondcited by 132
  8. 2025A Survey of Frontiers in LLM Reasoning: Inference Scaling, Learning to Reason, and Agentic Systemscited by 111
  9. 2025An Empirical Study on Eliciting and Improving R1-like Reasoning Modelscited by 78
  10. 2025Revisiting the Test-Time Scaling of o1-like Models: Do they Truly Possess Test-Time Scaling Capabilities?cited by 70
  11. 2025RedStar: Does Scaling Long-CoT Data Unlock Better Slow-Reasoning Systems?cited by 60
  12. 2025InftyThink: Breaking the Length Limits of Long-Context Reasoning in Large Language Modelscited by 54
38 more
  1. 2025MARFT: Multi-Agent Reinforcement Fine-Tuningcited by 52
  2. 2025MT-R1-Zero: Advancing LLM-based Machine Translation via R1-Zero-like Reinforcement Learningcited by 48
  3. 2025PonderLM: Pretraining Language Models to Ponder in Continuous Spacecited by 44
  4. 2025Deconstructing Long Chain-of-Thought: A Structured Reasoning Optimization Framework for Long CoT Distillationcited by 43
  5. 2025Reasoning Language Models: A Blueprintcited by 42
  6. 2025VisuoThink: Empowering LVLM Reasoning with Multimodal Tree Searchcited by 36
  7. 2025Beyond Distillation: Pushing the Limits of Medical LLM Reasoning with Minimalist Rule-Based RLcited by 36
  8. 2025Two Heads are Better Than One: Test-time Scaling of Multi-agent Collaborative Reasoningcited by 29
  9. 2025Incentivizing Dual Process Thinking for Efficient Large Language Model Reasoningcited by 21
  10. 2025A Survey of Slow Thinking-based Reasoning LLMs using Reinforced Learning and Inference-time Scaling Lawcited by 19
  11. 2025On the Effect of Sampling Diversity in Scaling LLM Inferencecited by 18
  12. 2025Direct Reasoning Optimization: Token-Level Reasoning Reflectivity Meets Rubric Gates for Unverifiable Taskscited by 18
  13. 2025Agent models: Internalizing Chain-of-Action Generation into Reasoning modelscited by 17
  14. 2025Learning from Peers in Reasoning Modelscited by 16
  15. 2025Generative AI Act II: Test Time Scaling Drives Cognition Engineeringcited by 15
  16. 2025Enhancing Latent Computation in Transformers with Latent Tokenscited by 14
  17. 2025Generating Symbolic World Models via Test-time Scaling of Large Language Modelscited by 12
  18. 2025A Survey on Feedback-based Multi-step Reasoning for Large Language Models on Mathematicscited by 11
  19. 2025MT-RewardTree: A Comprehensive Framework for Advancing LLM-Based Machine Translation via Reward Modelingcited by 10
  20. 2025DiRL: An Efficient Post-Training Framework for Diffusion Language Modelscited by 10
  21. 2025DISC: Dynamic Decomposition Improves LLM Inference Scalingcited by 9
  22. 2025J4R: Learning to Judge with Equivalent Initial State Group Relative Policy Optimizationcited by 9
  23. 2025Iterative Deepening Sampling as Efficient Test-Time Scalingcited by 8
  24. 2025RLTHF: Targeted Human Feedback for LLM Alignmentcited by 8
  25. 2025Beyond Accuracy: Dissecting Mathematical Reasoning for LLMs Under Reinforcement Learningcited by 7
  26. 2025TeaRAG: A Token-Efficient Agentic Retrieval-Augmented Generation Frameworkcited by 7
  27. 2025Knowledge Augmented Complex Problem Solving with Large Language Models: A Surveycited by 6
  28. 2025Inference Computation Scaling for Feature Augmentation in Recommendation Systemscited by 5
  29. 2025Surgery-R1: Advancing Surgical-VQLA with Reasoning Multimodal Large Language Model via Reinforcement Learningcited by 5
  30. 2025Fixing the Broken Compass: Diagnosing and Improving Inference-Time Reward Modelingcited by 4
  31. 2025KG-o1: Enhancing Multi-hop Question Answering in Large Language Models via Knowledge Graph Integrationcited by 4
  32. 2026Balanced Aggregation: Understanding and Fixing Aggregation Bias in GRPOcited by 3
  33. 2025Beware of Reasoning Overconfidence: Pitfalls in the Reasoning Process for Multi-solution Taskscited by 2
  34. 2025ARISE: An Adaptive Resolution-Aware Metric for Test-Time Scaling Evaluation in Large Reasoning Modelscited by 1
  35. 2026CAT: Confidence-Adaptive Thinking for Efficient Reasoning of Large Reasoning Modelscited by 1
  36. 2025TimeHC-RL: Temporal-aware Hierarchical Cognitive Reinforcement Learning for Enhancing LLMs' Social Intelligencecited by 0
  37. 2026When LLM Meets Tree Search: A Systematic View of Inference as Search in Large Language Modelscited by 0
  38. 2026Mechanics of Long-Context Hybrid Models Part 1.1: From Hybrid Attention to Hybrid Positioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026OASES: Outcome-Aligned Search-Evaluation Co-Training for Agentic Search
  2. 2026IG-Search: Step-Level Information Gain Rewards for Search-Augmented Reasoning
  3. 2024A Comparative Study on Reasoning Patterns of OpenAI's o1 Model
  4. 2024On The Planning Abilities of OpenAI's o1 Models: Feasibility, Optimality, and Generalizability
  5. 2026Beyond Outcome Rewards: Step-Level Self-Distilled Policy Optimization for Deep Search Agents
  6. 2025OpenReward: Learning to Reward Long-form Agentic Tasks via Reinforcement Learning
  7. 2026SD-Search: On-Policy Hindsight Self-Distillation for Search-Augmented Reasoning
  8. 2026Towards a Mechanistic Understanding of Large Reasoning Models: A Survey of Training, Inference, and Failures