# Marco-o1 v2: Towards Widening The Distillation Bottleneck for Reasoning Models

arXiv: https://arxiv.org/abs/2503.01461 · HTML: https://mlsanity.com/paper/2503.01461/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Placed in Prompting and chain-of-thought reasoning, one of 17 directions in LLM reasoning.
- Builds on 23 of its 33 references that are in the map.
- First posted on arXiv 3 Mar 2025.

## Placed in

- LLM reasoning / [Prompting and chain-of-thought reasoning](https://mlsanity.com/lr/prompting-and-chain-of-thought-reasoning/)

## Builds on (most cited first, 23 of 23)

- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [Let's Verify Step by Step](https://mlsanity.com/paper/2305.20050/) (2023)
- [Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotations](https://mlsanity.com/paper/2312.08935/) (2023)
- [Do NOT Think That Much for 2+3=? On the Overthinking of o1-Like LLMs](https://mlsanity.com/paper/2412.21187/) (2024)
- [Improve Mathematical Reasoning in Language Models by Automated Process Supervision](https://mlsanity.com/paper/2406.06592/) (2024)
- [PlanBench: An Extensible Benchmark for Evaluating Large Language Models on Planning and Reasoning about Change](https://mlsanity.com/paper/2206.10498/) (2022)
- [ReST-MCTS*: LLM Self-Training via Process Reward Guided Tree Search](https://mlsanity.com/paper/2406.03816/) (2024)
- [Specializing Smaller Language Models towards Multi-Step Reasoning](https://mlsanity.com/paper/2301.12726/) (2023)
- [Thoughts Are All Over the Place: On the Underthinking of o1-Like LLMs](https://mlsanity.com/paper/2501.18585/) (2025)
- [Marco-o1: Towards Open Reasoning Models for Open-Ended Solutions](https://mlsanity.com/paper/2411.14405/) (2024)
- [O1 Replication Journey: A Strategic Progress Report -- Part 1](https://mlsanity.com/paper/2410.18982/) (2024)
- [Mutual Reasoning Makes Smaller LLMs Stronger Problem-Solvers](https://mlsanity.com/paper/2408.06195/) (2024)
- [Evaluation of OpenAI o1: Opportunities and Challenges of AGI](https://mlsanity.com/paper/2409.18486/) (2024)
- [Toward Self-Improvement of LLMs via Imagination, Searching, and Criticizing](https://mlsanity.com/paper/2404.12253/) (2024)
- [Small Models Struggle to Learn from Strong Reasoners](https://mlsanity.com/paper/2502.12143/) (2025)
- [Multi-IF: Benchmarking LLMs on Multi-Turn and Multilingual Instructions Following](https://mlsanity.com/paper/2410.15553/) (2024)
- [O1 Replication Journey -- Part 2: Surpassing O1-preview through Simple Distillation, Big Progress or Bitter Lesson?](https://mlsanity.com/paper/2411.16489/) (2024)
- [Eliminating Biased Length Reliance of Direct Preference Optimization via Down-Sampled KL Divergence](https://mlsanity.com/paper/2406.10957/) (2024)
- [Understanding Forgetting in LLM Supervised Fine-Tuning and Preference Learning -- A Convex Optimization Perspective](https://mlsanity.com/paper/2410.15483/) (2024)
- [UFT: Unifying Fine-Tuning of SFT and RLHF/DPO/UNA through a Generalized Implicit Reward Function](https://mlsanity.com/paper/2410.21438/) (2024)
- [Length Desensitization in Direct Preference Optimization](https://mlsanity.com/paper/2409.06411/) (2024)

## Built on by (most cited first, 13 of 13)

- [WebSailor: Navigating Super-human Reasoning for Web Agent](https://mlsanity.com/paper/2507.02592/) (2025)
- [WebDancer: Towards Autonomous Information Seeking Agency](https://mlsanity.com/paper/2505.22648/) (2025)
- [Pangu Embedded: An Efficient Dual-system LLM Reasoner with Metacognition](https://mlsanity.com/paper/2505.22375/) (2025)
- [Trade-offs in Large Reasoning Models: An Empirical Analysis of Deliberative and Adaptive Reasoning over Foundational Capabilities](https://mlsanity.com/paper/2503.17979/) (2025)
- [QFFT, Question-Free Fine-Tuning for Adaptive Reasoning](https://mlsanity.com/paper/2506.12860/) (2025)
- [Revisiting Test-Time Scaling: A Survey and a Diversity-Aware Method for Efficient Reasoning](https://mlsanity.com/paper/2506.04611/) (2025)
- [MiCoTA: Bridging the Learnability Gap with Intermediate CoT and Teacher Assistants](https://mlsanity.com/paper/2507.01887/) (2025)
- [Long-Chain Reasoning Distillation via Adaptive Prefix Alignment](https://mlsanity.com/paper/2601.10064/) (2026)
- [Enhancing Long-Chain Reasoning Distillation through Error-Aware Self-Reflection](https://mlsanity.com/paper/2505.22131/) (2025)
- [Difficulty-Estimated Policy Optimization](https://mlsanity.com/paper/2602.06375/) (2026)
- [Lucy: edgerunning agentic web search on mobile with machine generated task vectors](https://mlsanity.com/paper/2508.00360/) (2025)
- [Distilling Long-CoT Reasoning through Collaborative Step-wise Multi-Teacher Decoding](https://mlsanity.com/paper/2605.02290/) (2026)
- [Discovering Millions of Interpretable Features with Sparse Autoencoders](https://mlsanity.com/paper/2606.26620/) (2026)

## Abstract (arXiv)

Large Reasoning Models(LRMs) such as OpenAI o1 and DeepSeek-R1 have shown remarkable reasoning capabilities by scaling test-time compute and generating long Chain-of-Thought(CoT). Distillation--post-training on LRMs-generated data--is a straightforward yet effective method to enhance the reasoning abilities of smaller models, but faces a critical bottleneck: we found that distilled long CoT data poses learning difficulty for small models and leads to the inheritance of biases (i.e. over-thinking) when using Supervised Fine-tuning (SFT) and Reinforcement Learning (RL) methods. To alleviate this bottleneck, we propose constructing tree-based CoT data from scratch via Monte Carlo Tree Search(MCTS). We then exploit a set of CoT-aware approaches, including Thoughts Length Balance, Fine-grained DPO, and Joint Post-training Objective, to enhance SFT and RL on the constructed data. We conduct evaluation on various benchmarks such as math (GSM8K, MATH, AIME). instruction-following (Multi-IF) and planning (Blocksworld), results demonstrate our approaches substantially improve the reasoning performance of distilled models compared to standard distilled models via reducing the hallucinations in long-time thinking. The project homepage is https://github.com/AIDC-AI/Marco-o1.