mlsanityData to 7 Oct 2026

Not All Correct Answers Are Equal: Why Your Distillation Source Matters

arXiv 2505.14464 ↗First posted 20 May 2025cs.CLcited by 25

On the map: LLM reasoning → Prompting and chain-of-thought reasoning

Abstract (arXiv)

Distillation has emerged as a practical and effective approach to enhance the reasoning capabilities of open-source language models. In this work, we conduct a large-scale empirical study on reasoning data distillation by collecting verified outputs from three state-of-the-art teacher models-AM-Thinking-v1, Qwen3-235B-A22B, and DeepSeek-R1-on a shared corpus of 1.89 million queries. We construct three parallel datasets and analyze their distributions, revealing that AM-Thinking-v1-distilled data exhibits greater token length diversity and lower perplexity. Student models trained on each dataset are evaluated on reasoning benchmarks including AIME2024, AIME2025, MATH500, and LiveCodeBench. The model distilled from AM-Thinking-v1 consistently achieves the best performance (e.g., 84.3 on AIME2024, 72.2 on AIME2025, 98.4 on MATH500, and 65.9 on LiveCodeBench) and demonstrates adaptive output behavior-producing longer responses for harder tasks and shorter ones for simpler tasks. These findings highlight the value of high-quality, verified reasoning traces. We release the AM-Thinking-v1 and Qwen3-235B-A22B distilled datasets to support future research on open and high-performing reasoning-oriented language models. The datasets are publicly available on Hugging FacefootnoteDatasets are available on Hugging Face: hrefhttps://huggingface.co/datasets/a-m-team/AM-Thinking-v1-DistilledAM-Thinking-v1-Distilled, hrefhttps://huggingface.co/datasets/a-m-team/AM-Qwen3-DistilledAM-Qwen3-Distilled..

Key facts. Cited by 25 papers in the map (data through 7 Oct 2026). Placed in Prompting and chain-of-thought reasoning, one of 17 directions in LLM reasoning. Builds on 8 of its 19 references that are in the map. First posted on arXiv 20 May 2025.

Builds on

8 of 19 references.

References · 8

  1. 2024Qwen2 Technical Reportcited by 2,076
  2. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 1,906
  3. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 825
  4. 2024Free Process Rewards without Process Labelscited by 162
  5. 2025KodCode: A Diverse, Challenging, and Verifiable Synthetic Dataset for Codingcited by 115
  6. 20251.4 Million Open-Source Distilled Reasoning Dataset to Empower Large Language Model Trainingcited by 49
  7. 2025AM-Thinking-v1: Advancing the Frontier of Reasoning at 32B Scalecited by 29
  8. 2025DeepDistill: Enhancing LLM Reasoning Capabilities via Large-Scale Difficulty-Graded Data Trainingcited by 17

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 11; 2026: 14.

25 papers.

Sort:

By area:

  1. 2025MiroThinker: Pushing the Performance Boundaries of Open-Source Research Agents via Model, Context, and Interactive Scalingcited by 72
  2. 2025ReCode: Reinforcing Code Generation with Reasoning-Process Rewardscited by 25
  3. 2025OpenDataArena: A Fair and Open Arena for Benchmarking Post-Training Dataset Valuecited by 15
  4. 2025K2-Think: A Parameter-Efficient Reasoning Systemcited by 11
  5. 2025KAT-Coder Technical Reportcited by 7
  6. 2025ScaleDiff: Scaling Difficult Problems for Advanced Mathematical Reasoningcited by 6
  7. 2025HiPO: Hybrid Policy Optimization for Dynamic Reasoning in LLMscited by 4
  8. 2025Closing the Data Loop: Using OpenDataArena to Engineer Superior Training Datasetscited by 4
  9. 2026On the Role of Reasoning Patterns in the Generalization Discrepancy of Long Chain-of-Thought Supervised Fine-Tuningcited by 4
  10. 2025A Comprehensive FP8 Training Recipe for Reasoning-Enhanced Language Modelscited by 3
  11. 2025Efficient Reasoning via Thought-Training and Thought-Free Inferencecited by 2
  12. 2026Tracing the Roots: A Multi-Agent Framework for Uncovering Data Lineage in Post-Training LLMscited by 2
13 more
  1. 2026When Perplexity Lies: Generation-Focused Distillation of Hybrid Sequence Modelscited by 1
  2. 2026SFT Conflicts, RL Coexists: A Theoretical and Empirical Analysis of Multi-Task Learning for LLMscited by 1
  3. 2026From Experts to Sub-experts: Fine-grained Parameter-Efficient Fine-Tuning for MoE LLMscited by 1
  4. 2025Direct Multi-Token Decodingcited by 0
  5. 2026Tailoring the Curriculum: Student-Centered Reasoning Distillation via Dynamic Data-Model Compatibilitycited by 0
  6. 2026Diagnosing Harmful Continuation in Answer-Correct Long-CoT Training Tracescited by 0
  7. 2026NebulaExp-8B: An Empirical Post-Training Pipeline via Full-Scale Ablation Researchcited by 0
  8. 2026CausalMix: Data Mixture as Causal Inference for Language Model Trainingcited by 0
  9. 2026Gold-Guided Programmatic Distillation for Financial Reasoning over Hybrid Tables and Textcited by 0
  10. 2026SANE: State Anomaly Neutralization for Stable Extreme-Context Delta-Rule Modelscited by 0
  11. 2026Online Self-Weighted Fine-Tuningcited by 0
  12. 2026Rufus-Air: An Open LLM Post-Training Recipecited by 0
  13. 2026MultiTalk: Scaling Full-Duplex Speech Models to Long, Multi-Party, Bilingual Conversationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Mixed Distillation Helps Smaller Language Model Better Reasoning
  2. 2023PaD: Program-aided Distillation Can Teach Small Models Reasoning Better than Chain-of-thought Fine-tuning
  3. 2024Improving Mathematical Reasoning Capabilities of Small Language Models via Feedback-Driven Distillation
  4. 2025UNDO: Understanding Distillation as Optimization
  5. 2026Distribution Corrected Offline Data Distillation for Large Language Models
  6. 2025Beyond Scaling Law: A Data-Efficient Distillation Framework for Reasoning
  7. 2024Knowledge Distillation Using Frontier Open-source LLMs: Generalizability and the Role of Synthetic Data
  8. 2025Distilling the Essence: Efficient Reasoning Distillation via Sequence Truncation