# NaturalThoughts: Selecting and Distilling Reasoning Traces for General Reasoning Tasks

arXiv: https://arxiv.org/abs/2507.01921 · HTML: https://mlsanity.com/paper/2507.01921/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- Placed in Prompting and chain-of-thought reasoning, one of 17 directions in LLM reasoning.
- Builds on 20 of its 27 references that are in the map.
- First posted on arXiv 2 Jul 2025.

## Placed in

- LLM reasoning / [Prompting and chain-of-thought reasoning](https://mlsanity.com/lr/prompting-and-chain-of-thought-reasoning/)
- Knowledge distillation / [Knowledge distillation for LLMs](https://mlsanity.com/distill/knowledge-distillation-for-llms/)

## Builds on (most cited first, 20 of 20)

- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [Measuring Mathematical Problem Solving With the MATH Dataset](https://mlsanity.com/paper/2103.03874/) (2021)
- [GPQA: A Graduate-Level Google-Proof Q&A Benchmark](https://mlsanity.com/paper/2311.12022/) (2023)
- [SimCSE: Simple Contrastive Learning of Sentence Embeddings](https://mlsanity.com/paper/2104.08821/) (2021)
- [MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark](https://mlsanity.com/paper/2406.01574/) (2024)
- [MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers](https://mlsanity.com/paper/2002.10957/) (2020)
- [Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?](https://mlsanity.com/paper/2504.13837/) (2025)
- [On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes](https://mlsanity.com/paper/2306.13649/) (2023)
- [Do NOT Think That Much for 2+3=? On the Overthinking of o1-Like LLMs](https://mlsanity.com/paper/2412.21187/) (2024)
- [LIMO: Less is More for Reasoning](https://mlsanity.com/paper/2502.03387/) (2025)
- [L1: Controlling How Long A Reasoning Model Thinks With Reinforcement Learning](https://mlsanity.com/paper/2503.04697/) (2025)
- [OpenThoughts: Data Recipes for Reasoning Models](https://mlsanity.com/paper/2506.04178/) (2025)
- [Reasoning Models Can Be Effective Without Thinking](https://mlsanity.com/paper/2504.09858/) (2025)
- [Thoughts Are All Over the Place: On the Underthinking of o1-Like LLMs](https://mlsanity.com/paper/2501.18585/) (2025)
- [Distilling System 2 into System 1](https://mlsanity.com/paper/2407.06023/) (2024)
- [OverThink: Slowdown Attacks on Reasoning LLMs](https://mlsanity.com/paper/2502.02542/) (2025)
- [Long Is More for Alignment: A Simple but Tough-to-Beat Baseline for Instruction Fine-Tuning](https://mlsanity.com/paper/2402.04833/) (2024)
- [NaturalReasoning: Reasoning in the Wild with 2.8M Challenging Questions](https://mlsanity.com/paper/2502.13124/) (2025)
- [Retro-Search: Exploring Untaken Paths for Deeper and Efficient Reasoning](https://mlsanity.com/paper/2504.04383/) (2025)
- [Climbing the Ladder of Reasoning: What LLMs Can-and Still Can't-Solve after SFT?](https://mlsanity.com/paper/2504.11741/) (2025)

## Built on by (most cited first, 30 of 32)

- [MegaScience: Pushing the Frontiers of Post-Training Datasets for Science Reasoning](https://mlsanity.com/paper/2507.16812/) (2025)
- [Quagmires in SFT-RL Post-Training: When High SFT Scores Mislead and What to Use Instead](https://mlsanity.com/paper/2510.01624/) (2025)
- [Learning to Reason for Factuality](https://mlsanity.com/paper/2508.05618/) (2025)
- [Distribution-Aligned Sequence Distillation for Superior Long-CoT Reasoning](https://mlsanity.com/paper/2601.09088/) (2026)
- [Proximal Supervised Fine-Tuning](https://mlsanity.com/paper/2508.17784/) (2025)
- [From Code Foundation Models to Agents and Applications: A Comprehensive Survey and Practical Guide to Code Intelligence](https://mlsanity.com/paper/2511.18538/) (2025)
- [Executable Counterfactuals: Improving LLMs' Causal Reasoning Through Code](https://mlsanity.com/paper/2510.01539/) (2025)
- [Backtracking When It Strays: Mitigating Dual Exposure Biases in LLM Reasoning Distillation](https://mlsanity.com/paper/2605.19433/) (2026)
- [Unlocking Reasoning Capability on Machine Translation in Large Language Models](https://mlsanity.com/paper/2602.14763/) (2026)
- [Learning from Synthetic Data Improves Multi-hop Reasoning](https://mlsanity.com/paper/2603.02091/) (2026)
- [Distilling the Essence: Efficient Reasoning Distillation via Sequence Truncation](https://mlsanity.com/paper/2512.21002/) (2025)
- [Skill-Aware Data Selection and Fine-Tuning for Data-Efficient Reasoning Distillation](https://mlsanity.com/paper/2601.10109/) (2026)
- [When Models Judge Themselves: Unsupervised Self-Evolution for Multimodal Reasoning](https://mlsanity.com/paper/2603.21289/) (2026)
- [MR-Align: Meta-Reasoning Informed Factuality Alignment for Large Reasoning Models](https://mlsanity.com/paper/2510.24794/) (2025)
- [Tailored Primitive Initialization is the Secret Key to Reinforcement Learning](https://mlsanity.com/paper/2511.12429/) (2025)
- [A Critical Look at Targeted Instruction Selection: Disentangling What Matters (and What Doesn't)](https://mlsanity.com/paper/2602.14696/) (2026)
- [Multi-Agent Transactive Memory](https://mlsanity.com/paper/2606.19911/) (2026)
- [Data-Centric Lessons To Improve Speech-Language Pretraining](https://mlsanity.com/paper/2510.20860/) (2025)
- [AIR: Post-training Data Selection for Reasoning via Attention Head Influence](https://mlsanity.com/paper/2512.13279/) (2025)
- [Stepwise Penalization for Length-Efficient Chain-of-Thought Reasoning](https://mlsanity.com/paper/2603.00296/) (2026)
- [SightSound-R1: Cross-Modal Reasoning Distillation from Vision to Audio Language Models](https://mlsanity.com/paper/2509.15661/) (2025)
- [EtCon: Edit-then-Consolidate for Reliable Knowledge Editing](https://mlsanity.com/paper/2512.04753/) (2025)
- [STOP: Structured On-Policy Pruning of Long-Form Reasoning in Low-Data Regimes](https://mlsanity.com/paper/2605.13165/) (2026)
- [Metacognitive Behavioral Tuning of Large Language Models for Multi-Hop Question Answering](https://mlsanity.com/paper/2602.22508/) (2026)
- [Characterizing Model-Native Skills](https://mlsanity.com/paper/2604.17614/) (2026)
- [Distilling Long-CoT Reasoning through Collaborative Step-wise Multi-Teacher Decoding](https://mlsanity.com/paper/2605.02290/) (2026)
- [Long-Context Reasoning Through Proxy-Based Chain-of-Thought Tuning](https://mlsanity.com/paper/2605.20201/) (2026)
- [Different Teachers, Different Capabilities: Sub-1B On-Device Distillation for Structured Text Enrichment](https://mlsanity.com/paper/2607.08268/) (2026)
- [Selecting Diverse SFT Traces Improves Post-RL Generalization](https://mlsanity.com/paper/2609.33780/) (2026)
- [ImproveAnyTask: An Autonomous Post-Training Harness for Iterative Model Self-Improvement](https://mlsanity.com/paper/2610.06347/) (2026)

## Abstract (arXiv)

Recent work has shown that distilling reasoning traces from a larger teacher model via supervised finetuning outperforms reinforcement learning with the smaller student model alone (Guo et al. 2025). However, there has not been a systematic study of what kind of reasoning demonstrations from the teacher are most effective in improving the student model's reasoning capabilities. In this work we curate high-quality "NaturalThoughts" by selecting reasoning traces from a strong teacher model based on a large pool of questions from NaturalReasoning (Yuan et al. 2025). We first conduct a systematic analysis of factors that affect distilling reasoning capabilities, in terms of sample efficiency and scalability for general reasoning tasks. We observe that simply scaling up data size with random sampling is a strong baseline with steady performance gains. Further, we find that selecting difficult examples that require more diverse reasoning strategies is more sample-efficient to transfer the teacher model's reasoning skills. Evaluated on both Llama and Qwen models, training with NaturalThoughts outperforms existing reasoning datasets such as OpenThoughts, LIMO, etc. on general STEM reasoning benchmarks including GPQA-Diamond, MMLU-Pro and SuperGPQA.