# o1-Coder: an o1 Replication for Coding

arXiv: https://arxiv.org/abs/2412.00154 · HTML: https://mlsanity.com/paper/2412.00154/

## Key facts

- Cited by 78 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 38 references that are in the map.
- First posted on arXiv 29 Nov 2024.

## Builds on (most cited first, 21 of 21)

- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Program Synthesis with Large Language Models](https://mlsanity.com/paper/2108.07732/) (2021)
- [QLoRA: Efficient Finetuning of Quantized LLMs](https://mlsanity.com/paper/2305.14314/) (2023)
- [Fine-Tuning Language Models from Human Preferences](https://mlsanity.com/paper/1909.08593/) (2019)
- [Qwen2 Technical Report](https://mlsanity.com/paper/2407.10671/) (2024)
- [DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligence](https://mlsanity.com/paper/2401.14196/) (2024)
- [Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotations](https://mlsanity.com/paper/2312.08935/) (2023)
- [Quiet-STaR: Language Models Can Teach Themselves to Think Before Speaking](https://mlsanity.com/paper/2403.09629/) (2024)
- [Alphazero-like Tree-Search can Guide Large Language Model Decoding and Training](https://mlsanity.com/paper/2309.17179/) (2023)
- [Mobile-Agent: Autonomous Multi-Modal Mobile Device Agent with Visual Perception](https://mlsanity.com/paper/2401.16158/) (2024)
- [Grounding Large Language Models in Interactive Environments with Online Reinforcement Learning](https://mlsanity.com/paper/2302.02662/) (2023)
- [Mobile-Agent-v2: Mobile Device Operation Assistant with Effective Navigation via Multi-Agent Collaboration](https://mlsanity.com/paper/2406.01014/) (2024)
- [TACO: Topics in Algorithmic COde generation dataset](https://mlsanity.com/paper/2312.14852/) (2023)
- [Marco-o1: Towards Open Reasoning Models for Open-Ended Solutions](https://mlsanity.com/paper/2411.14405/) (2024)
- [O1 Replication Journey: A Strategic Progress Report -- Part 1](https://mlsanity.com/paper/2410.18982/) (2024)
- [Mutual Reasoning Makes Smaller LLMs Stronger Problem-Solvers](https://mlsanity.com/paper/2408.06195/) (2024)
- [OpenR: An Open Source Framework for Advanced Reasoning with Large Language Models](https://mlsanity.com/paper/2410.09671/) (2024)
- [LLaMA-Berry: Pairwise Optimization for O1-like Olympiad-Level Mathematical Reasoning](https://mlsanity.com/paper/2410.02884/) (2024)
- [ALR2: A Retrieve-then-Reason Framework for Long-context Question Answering](https://mlsanity.com/paper/2410.03227/) (2024)
- [Thinking Fast and Slow in AI: the Role of Metacognition](https://mlsanity.com/paper/2110.01834/) (2021)
- [Equitable Access to Justice: Logical LLMs Show Promise](https://mlsanity.com/paper/2410.09904/) (2024)

## Built on by (most cited first, 30 of 78)

- [s1: Simple test-time scaling](https://mlsanity.com/paper/2501.19393/) (2025)
- [Visual-RFT: Visual Reinforcement Fine-Tuning](https://mlsanity.com/paper/2503.01785/) (2025)
- [Search-o1: Agentic Search-Enhanced Large Reasoning Models](https://mlsanity.com/paper/2501.05366/) (2025)
- [Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Models](https://mlsanity.com/paper/2503.09567/) (2025)
- [From System 1 to System 2: A Survey of Reasoning Large Language Models](https://mlsanity.com/paper/2502.17419/) (2025)
- [HuatuoGPT-o1, Towards Medical Complex Reasoning with LLMs](https://mlsanity.com/paper/2412.18925/) (2024)
- [A Survey on Test-Time Scaling in Large Language Models: What, How, Where, and How Well?](https://mlsanity.com/paper/2503.24235/) (2025)
- [The Landscape of Agentic Reinforcement Learning for LLMs: A Survey](https://mlsanity.com/paper/2509.02547/) (2025)
- [Q-Insight: Understanding Image Quality via Visual Reinforcement Learning](https://mlsanity.com/paper/2503.22679/) (2025)
- [GenPRM: Scaling Test-Time Compute of Process Reward Models via Generative Reasoning](https://mlsanity.com/paper/2504.00891/) (2025)
- [Audio-Reasoner: Improving Reasoning Capability in Large Audio Language Models](https://mlsanity.com/paper/2503.02318/) (2025)
- [Unified Multimodal Chain-of-Thought Reward Model through Reinforcement Fine-Tuning](https://mlsanity.com/paper/2505.03318/) (2025)
- [UniVG-R1: Reasoning Guided Universal Visual Grounding with Reinforcement Learning](https://mlsanity.com/paper/2505.14231/) (2025)
- [VLN-R1: Vision-Language Navigation via Reinforcement Fine-Tuning](https://mlsanity.com/paper/2506.17221/) (2025)
- [RedStar: Does Scaling Long-CoT Data Unlock Better Slow-Reasoning Systems?](https://mlsanity.com/paper/2501.11284/) (2025)
- [Co-Evolving LLM Coder and Unit Tester via Reinforcement Learning](https://mlsanity.com/paper/2506.03136/) (2025)
- [Logical Reasoning in Large Language Models: A Survey](https://mlsanity.com/paper/2502.09100/) (2025)
- [Does Thinking More always Help? Mirage of Test-Time Scaling in Reasoning Models](https://mlsanity.com/paper/2506.04210/) (2025)
- [Embodied-Reasoner: Synergizing Visual Search, Reasoning, and Action for Embodied Interactive Tasks](https://mlsanity.com/paper/2503.21696/) (2025)
- [A Survey of Test-Time Compute: From Intuitive Inference to Deliberate Reasoning](https://mlsanity.com/paper/2501.02497/) (2025)
- [JarvisArt: Liberating Human Artistic Creativity via an Intelligent Photo Retouching Agent](https://mlsanity.com/paper/2506.17612/) (2025)
- [O1 Replication Journey -- Part 3: Inference-time Scaling for Medical Reasoning](https://mlsanity.com/paper/2501.06458/) (2025)
- [Search and Refine During Think: Facilitating Knowledge Refinement for Improved Retrieval-Augmented Reasoning](https://mlsanity.com/paper/2505.11277/) (2025)
- [Scalable Chain of Thoughts via Elastic Reasoning](https://mlsanity.com/paper/2505.05315/) (2025)
- [DRT: Deep Reasoning Translation via Long Chain-of-Thought](https://mlsanity.com/paper/2412.17498/) (2024)
- [Two Heads are Better Than One: Test-time Scaling of Multi-agent Collaborative Reasoning](https://mlsanity.com/paper/2504.09772/) (2025)
- [video-SALMONN-o1: Reasoning-enhanced Audio-visual Large Language Model](https://mlsanity.com/paper/2502.11775/) (2025)
- [OptimAI: Optimization from Natural Language Using LLM-Powered AI Agents](https://mlsanity.com/paper/2504.16918/) (2025)
- [R1-T1: Fully Incentivizing Translation Capability in LLMs via Reasoning Learning](https://mlsanity.com/paper/2502.19735/) (2025)
- [DiagnosisArena: Benchmarking Diagnostic Reasoning for Large Language Models](https://mlsanity.com/paper/2505.14107/) (2025)

## Abstract (arXiv)

The technical report introduces O1-CODER, an attempt to replicate OpenAI's o1 model with a focus on coding tasks. It integrates reinforcement learning (RL) and Monte Carlo Tree Search (MCTS) to enhance the model's System-2 thinking capabilities. The framework includes training a Test Case Generator (TCG) for standardized code testing, using MCTS to generate code data with reasoning processes, and iteratively fine-tuning the policy model to initially produce pseudocode and then generate the full code. The report also addresses the opportunities and challenges in deploying o1-like models in real-world applications, suggesting transitioning to the System-2 paradigm and highlighting the imperative for world model construction. Updated model progress and experimental results will be reported in subsequent versions. All source code, curated datasets, as well as the derived models are disclosed at https://github.com/ADaM-BJTU/O1-CODER .