mlsanityData to 7 Oct 2026

NaturalThoughts: Selecting and Distilling Reasoning Traces for General Reasoning Tasks

arXiv 2507.01921 ↗First posted 2 Jul 2025cs.CLcited by 29

On the map: LLM reasoning → Prompting and chain-of-thought reasoning

Abstract (arXiv)

Recent work has shown that distilling reasoning traces from a larger teacher model via supervised finetuning outperforms reinforcement learning with the smaller student model alone (Guo et al. 2025). However, there has not been a systematic study of what kind of reasoning demonstrations from the teacher are most effective in improving the student model's reasoning capabilities. In this work we curate high-quality "NaturalThoughts" by selecting reasoning traces from a strong teacher model based on a large pool of questions from NaturalReasoning (Yuan et al. 2025). We first conduct a systematic analysis of factors that affect distilling reasoning capabilities, in terms of sample efficiency and scalability for general reasoning tasks. We observe that simply scaling up data size with random sampling is a strong baseline with steady performance gains. Further, we find that selecting difficult examples that require more diverse reasoning strategies is more sample-efficient to transfer the teacher model's reasoning skills. Evaluated on both Llama and Qwen models, training with NaturalThoughts outperforms existing reasoning datasets such as OpenThoughts, LIMO, etc. on general STEM reasoning benchmarks including GPQA-Diamond, MMLU-Pro and SuperGPQA.

Key facts. Cited by 29 papers in the map (data through 7 Oct 2026). Placed in Prompting and chain-of-thought reasoning, one of 17 directions in LLM reasoning. Builds on 20 of its 27 references that are in the map. First posted on arXiv 2 Jul 2025.

Builds on

20 of 27 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Distilling System 2 into System 18 Jul

Other references · 19

  1. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,146
  2. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  3. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,748
  4. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,168
  5. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,758
  6. 2020MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformerscited by 1,414
  7. 2025Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?cited by 948
  8. 2023On-Policy Distillation of Language Models: Learning from Self-Generated Mistakescited by 840
  9. 2024Do NOT Think That Much for 2+3=? On the Overthinking of o1-Like LLMscited by 564
  10. 2025LIMO: Less is More for Reasoningcited by 488
  11. 2025L1: Controlling How Long A Reasoning Model Thinks With Reinforcement Learningcited by 381
  12. 2025OpenThoughts: Data Recipes for Reasoning Modelscited by 287
7 more
  1. 2025Reasoning Models Can Be Effective Without Thinkingcited by 182
  2. 2025Thoughts Are All Over the Place: On the Underthinking of o1-Like LLMscited by 162
  3. 2025OverThink: Slowdown Attacks on Reasoning LLMscited by 94
  4. 2024Long Is More for Alignment: A Simple but Tough-to-Beat Baseline for Instruction Fine-Tuningcited by 85
  5. 2025NaturalReasoning: Reasoning in the Wild with 2.8M Challenging Questionscited by 81
  6. 2025Retro-Search: Exploring Untaken Paths for Deeper and Efficient Reasoningcited by 32
  7. 2025Climbing the Ladder of Reasoning: What LLMs Can-and Still Can't-Solve after SFT?cited by 21

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 13; 2026: 16.

29 papers.

Sort:

By area:

  1. 2025MegaScience: Pushing the Frontiers of Post-Training Datasets for Science Reasoningcited by 56
  2. 2025Quagmires in SFT-RL Post-Training: When High SFT Scores Mislead and What to Use Insteadcited by 24
  3. 2025Learning to Reason for Factualitycited by 22
  4. 2025Proximal Supervised Fine-Tuningcited by 15
  5. 2025From Code Foundation Models to Agents and Applications: A Comprehensive Survey and Practical Guide to Code Intelligencecited by 13
  6. 2026Distribution-Aligned Sequence Distillation for Superior Long-CoT Reasoningcited by 13
  7. 2025Executable Counterfactuals: Improving LLMs' Causal Reasoning Through Codecited by 10
  8. 2026Unlocking Reasoning Capability on Machine Translation in Large Language Modelscited by 6
  9. 2026Learning from Synthetic Data Improves Multi-hop Reasoningcited by 6
  10. 2025Distilling the Essence: Efficient Reasoning Distillation via Sequence Truncationcited by 5
  11. 2026Backtracking When It Strays: Mitigating Dual Exposure Biases in LLM Reasoning Distillationcited by 5
  12. 2025MR-Align: Meta-Reasoning Informed Factuality Alignment for Large Reasoning Modelscited by 4
17 more
  1. 2025Tailored Primitive Initialization is the Secret Key to Reinforcement Learningcited by 4
  2. 2026Skill-Aware Data Selection and Fine-Tuning for Data-Efficient Reasoning Distillationcited by 4
  3. 2026When Models Judge Themselves: Unsupervised Self-Evolution for Multimodal Reasoningcited by 4
  4. 2026A Critical Look at Targeted Instruction Selection: Disentangling What Matters (and What Doesn't)cited by 3
  5. 2026Stepwise Penalization for Length-Efficient Chain-of-Thought Reasoningcited by 3
  6. 2025Data-Centric Lessons To Improve Speech-Language Pretrainingcited by 2
  7. 2025AIR: Post-training Data Selection for Reasoning via Attention Head Influencecited by 2
  8. 2025SightSound-R1: Cross-Modal Reasoning Distillation from Vision to Audio Language Modelscited by 1
  9. 2025EtCon: Edit-then-Consolidate for Reliable Knowledge Editingcited by 1
  10. 2026Multi-Agent Transactive Memorycited by 1
  11. 2026Metacognitive Behavioral Tuning of Large Language Models for Multi-Hop Question Answeringcited by 0
  12. 2026Characterizing Model-Native Skillscited by 0
  13. 2026Distilling Long-CoT Reasoning through Collaborative Step-wise Multi-Teacher Decodingcited by 0
  14. 2026STOP: Structured On-Policy Pruning of Long-Form Reasoning in Low-Data Regimescited by 0
  15. 2026Long-Context Reasoning Through Proxy-Based Chain-of-Thought Tuningcited by 0
  16. 2026Different Teachers, Different Capabilities: Sub-1B On-Device Distillation for Structured Text Enrichmentcited by 0
  17. 2026Selecting Diverse SFT Traces Improves Post-RL Generalizationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Harnessing Negative Signals: Reinforcement Distillation from Teacher Data for LLM Reasoning
  2. 2025Concise Reasoning, Big Gains: Pruning Long Reasoning Trace with Difficulty-Aware Prompting
  3. 2026DeReason: A Difficulty-Aware Curriculum Improves Decoupled SFT-then-RL Training for General Reasoning
  4. 2025Beyond Scaling Law: A Data-Efficient Distillation Framework for Reasoning
  5. 2025The Signal is in the Steps: Local Scoring for Reasoning Data Selection
  6. 2025RLKD: Distilling LLMs' Reasoning via Reinforcement Learning
  7. 2025Scale or Reason? A Compute-Equivalent Analysis of Reasoning Distillation
  8. 2026Distribution Corrected Offline Data Distillation for Large Language Models