# Not All Correct Answers Are Equal: Why Your Distillation Source Matters

arXiv: https://arxiv.org/abs/2505.14464 · HTML: https://mlsanity.com/paper/2505.14464/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- Placed in Prompting and chain-of-thought reasoning, one of 17 directions in LLM reasoning.
- Builds on 8 of its 19 references that are in the map.
- First posted on arXiv 20 May 2025.

## Placed in

- LLM reasoning / [Prompting and chain-of-thought reasoning](https://mlsanity.com/lr/prompting-and-chain-of-thought-reasoning/)

## Builds on (most cited first, 8 of 8)

- [Qwen2 Technical Report](https://mlsanity.com/paper/2407.10671/) (2024)
- [LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code](https://mlsanity.com/paper/2403.07974/) (2024)
- [Enhancing Chat Language Models by Scaling High-quality Instructional Conversations](https://mlsanity.com/paper/2305.14233/) (2023)
- [Free Process Rewards without Process Labels](https://mlsanity.com/paper/2412.01981/) (2024)
- [KodCode: A Diverse, Challenging, and Verifiable Synthetic Dataset for Coding](https://mlsanity.com/paper/2503.02951/) (2025)
- [1.4 Million Open-Source Distilled Reasoning Dataset to Empower Large Language Model Training](https://mlsanity.com/paper/2503.19633/) (2025)
- [AM-Thinking-v1: Advancing the Frontier of Reasoning at 32B Scale](https://mlsanity.com/paper/2505.08311/) (2025)
- [DeepDistill: Enhancing LLM Reasoning Capabilities via Large-Scale Difficulty-Graded Data Training](https://mlsanity.com/paper/2504.17565/) (2025)

## Built on by (most cited first, 25 of 25)

- [MiroThinker: Pushing the Performance Boundaries of Open-Source Research Agents via Model, Context, and Interactive Scaling](https://mlsanity.com/paper/2511.11793/) (2025)
- [ReCode: Reinforcing Code Generation with Reasoning-Process Rewards](https://mlsanity.com/paper/2508.05170/) (2025)
- [OpenDataArena: A Fair and Open Arena for Benchmarking Post-Training Dataset Value](https://mlsanity.com/paper/2512.14051/) (2025)
- [K2-Think: A Parameter-Efficient Reasoning System](https://mlsanity.com/paper/2509.07604/) (2025)
- [KAT-Coder Technical Report](https://mlsanity.com/paper/2510.18779/) (2025)
- [ScaleDiff: Scaling Difficult Problems for Advanced Mathematical Reasoning](https://mlsanity.com/paper/2509.21070/) (2025)
- [HiPO: Hybrid Policy Optimization for Dynamic Reasoning in LLMs](https://mlsanity.com/paper/2509.23967/) (2025)
- [Closing the Data Loop: Using OpenDataArena to Engineer Superior Training Datasets](https://mlsanity.com/paper/2601.09733/) (2025)
- [On the Role of Reasoning Patterns in the Generalization Discrepancy of Long Chain-of-Thought Supervised Fine-Tuning](https://mlsanity.com/paper/2604.01702/) (2026)
- [A Comprehensive FP8 Training Recipe for Reasoning-Enhanced Language Models](https://mlsanity.com/paper/2509.22536/) (2025)
- [Efficient Reasoning via Thought-Training and Thought-Free Inference](https://mlsanity.com/paper/2511.03408/) (2025)
- [Tracing the Roots: A Multi-Agent Framework for Uncovering Data Lineage in Post-Training LLMs](https://mlsanity.com/paper/2604.10480/) (2026)
- [Tailoring the Curriculum: Student-Centered Reasoning Distillation via Dynamic Data-Model Compatibility](https://mlsanity.com/paper/2605.29229/) (2026)
- [SFT Conflicts, RL Coexists: A Theoretical and Empirical Analysis of Multi-Task Learning for LLMs](https://mlsanity.com/paper/2608.03573/) (2026)
- [When Perplexity Lies: Generation-Focused Distillation of Hybrid Sequence Models](https://mlsanity.com/paper/2603.26556/) (2026)
- [From Experts to Sub-experts: Fine-grained Parameter-Efficient Fine-Tuning for MoE LLMs](https://mlsanity.com/paper/2609.25655/) (2026)
- [MultiTalk: Scaling Full-Duplex Speech Models to Long, Multi-Party, Bilingual Conversation](https://mlsanity.com/paper/2609.36903/) (2026)
- [Direct Multi-Token Decoding](https://mlsanity.com/paper/2510.11958/) (2025)
- [Diagnosing Harmful Continuation in Answer-Correct Long-CoT Training Traces](https://mlsanity.com/paper/2605.29288/) (2026)
- [NebulaExp-8B: An Empirical Post-Training Pipeline via Full-Scale Ablation Research](https://mlsanity.com/paper/2606.26671/) (2026)
- [CausalMix: Data Mixture as Causal Inference for Language Model Training](https://mlsanity.com/paper/2607.01104/) (2026)
- [Gold-Guided Programmatic Distillation for Financial Reasoning over Hybrid Tables and Text](https://mlsanity.com/paper/2607.14709/) (2026)
- [SANE: State Anomaly Neutralization for Stable Extreme-Context Delta-Rule Models](https://mlsanity.com/paper/2608.22354/) (2026)
- [Online Self-Weighted Fine-Tuning](https://mlsanity.com/paper/2609.00734/) (2026)
- [Rufus-Air: An Open LLM Post-Training Recipe](https://mlsanity.com/paper/2609.29421/) (2026)

## Abstract (arXiv)

Distillation has emerged as a practical and effective approach to enhance the reasoning capabilities of open-source language models. In this work, we conduct a large-scale empirical study on reasoning data distillation by collecting verified outputs from three state-of-the-art teacher models-AM-Thinking-v1, Qwen3-235B-A22B, and DeepSeek-R1-on a shared corpus of 1.89 million queries. We construct three parallel datasets and analyze their distributions, revealing that AM-Thinking-v1-distilled data exhibits greater token length diversity and lower perplexity. Student models trained on each dataset are evaluated on reasoning benchmarks including AIME2024, AIME2025, MATH500, and LiveCodeBench. The model distilled from AM-Thinking-v1 consistently achieves the best performance (e.g., 84.3 on AIME2024, 72.2 on AIME2025, 98.4 on MATH500, and 65.9 on LiveCodeBench) and demonstrates adaptive output behavior-producing longer responses for harder tasks and shorter ones for simpler tasks. These findings highlight the value of high-quality, verified reasoning traces. We release the AM-Thinking-v1 and Qwen3-235B-A22B distilled datasets to support future research on open and high-performing reasoning-oriented language models. The datasets are publicly available on Hugging FacefootnoteDatasets are available on Hugging Face: hrefhttps://huggingface.co/datasets/a-m-team/AM-Thinking-v1-DistilledAM-Thinking-v1-Distilled, hrefhttps://huggingface.co/datasets/a-m-team/AM-Qwen3-DistilledAM-Qwen3-Distilled..