# 1.4 Million Open-Source Distilled Reasoning Dataset to Empower Large Language Model Training

arXiv: https://arxiv.org/abs/2503.19633 · HTML: https://mlsanity.com/paper/2503.19633/

## Key facts

- Cited by 49 papers in the map (data through 14 Oct 2026).
- Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning.
- Builds on 17 of its 29 references that are in the map.
- First posted on arXiv 25 Mar 2025.

## Placed in

- LLM reasoning / [Reasoning benchmarks and evaluation](https://mlsanity.com/lr/reasoning-benchmarks-and-evaluation/)
- Knowledge distillation / [Knowledge distillation for LLMs](https://mlsanity.com/distill/knowledge-distillation-for-llms/)

## Builds on (most cited first, 17 of 17)

- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Measuring Mathematical Problem Solving With the MATH Dataset](https://mlsanity.com/paper/2103.03874/) (2021)
- [GPQA: A Graduate-Level Google-Proof Q&A Benchmark](https://mlsanity.com/paper/2311.12022/) (2023)
- [LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code](https://mlsanity.com/paper/2403.07974/) (2024)
- [Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters](https://mlsanity.com/paper/2408.03314/) (2024)
- [s1: Simple test-time scaling](https://mlsanity.com/paper/2501.19393/) (2025)
- [MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Models](https://mlsanity.com/paper/2309.12284/) (2023)
- [LIMO: Less is More for Reasoning](https://mlsanity.com/paper/2502.03387/) (2025)
- [Inference Scaling Laws: An Empirical Analysis of Compute-Optimal Inference for Problem-Solving with Language Models](https://mlsanity.com/paper/2408.00724/) (2024)
- [Free Process Rewards without Process Labels](https://mlsanity.com/paper/2412.01981/) (2024)
- [OpenCoder: The Open Cookbook for Top-Tier Code Large Language Models](https://mlsanity.com/paper/2411.04905/) (2024)
- [KodCode: A Diverse, Challenging, and Verifiable Synthetic Dataset for Coding](https://mlsanity.com/paper/2503.02951/) (2025)
- [Synthetic Data Generation with Large Language Models for Text Classification: Potential and Limitations](https://mlsanity.com/paper/2310.07849/) (2023)
- [NaturalReasoning: Reasoning in the Wild with 2.8M Challenging Questions](https://mlsanity.com/paper/2502.13124/) (2025)
- [CodeI/O: Condensing Reasoning Patterns via Code Input-Output Prediction](https://mlsanity.com/paper/2502.07316/) (2025)
- [Self-Explore: Enhancing Mathematical Reasoning in Language Models with Fine-grained Rewards](https://mlsanity.com/paper/2404.10346/) (2024)
- [Scaling Data Diversity for Fine-Tuning Language Models in Human Alignment](https://mlsanity.com/paper/2403.11124/) (2024)

## Built on by (most cited first, 30 of 49)

- [AIMO-2 Winning Solution: Building State-of-the-Art Mathematical Reasoning Models with OpenMathReasoning dataset](https://mlsanity.com/paper/2504.16891/) (2025)
- [Think Only When You Need with Large Hybrid-Reasoning Models](https://mlsanity.com/paper/2505.14631/) (2025)
- [Beyond Chemical QA: Evaluating LLM's Chemical Reasoning with Modular Chemical Operations](https://mlsanity.com/paper/2505.21318/) (2025)
- [SwS: Self-aware Weakness-driven Problem Synthesis in Reinforcement Learning for LLM Reasoning](https://mlsanity.com/paper/2506.08989/) (2025)
- [Finetuning LLMs for Human Behavior Prediction in Social Science Experiments](https://mlsanity.com/paper/2509.05830/) (2025)
- [100 Days After DeepSeek-R1: A Survey on Replication Studies and More Directions for Reasoning Language Models](https://mlsanity.com/paper/2505.00551/) (2025)
- [AM-Thinking-v1: Advancing the Frontier of Reasoning at 32B Scale](https://mlsanity.com/paper/2505.08311/) (2025)
- [Nemotron-Math: Efficient Long-Context Distillation of Mathematical Reasoning from Multi-Mode Supervision](https://mlsanity.com/paper/2512.15489/) (2025)
- [Not All Correct Answers Are Equal: Why Your Distillation Source Matters](https://mlsanity.com/paper/2505.14464/) (2025)
- [Outcome-based Reinforcement Learning to Predict the Future](https://mlsanity.com/paper/2505.17989/) (2025)
- [Not All Tokens Are What You Need In Thinking](https://mlsanity.com/paper/2505.17827/) (2025)
- [ChemDFM-R: A Chemical Reasoning LLM Enhanced with Atomized Chemical Knowledge](https://mlsanity.com/paper/2507.21990/) (2025)
- [DeepDistill: Enhancing LLM Reasoning Capabilities via Large-Scale Difficulty-Graded Data Training](https://mlsanity.com/paper/2504.17565/) (2025)
- [Towards Concise and Adaptive Thinking in Large Reasoning Models: A Survey](https://mlsanity.com/paper/2507.09662/) (2025)
- [Distribution-Aligned Sequence Distillation for Superior Long-CoT Reasoning](https://mlsanity.com/paper/2601.09088/) (2026)
- [Concise Reasoning, Big Gains: Pruning Long Reasoning Trace with Difficulty-Aware Prompting](https://mlsanity.com/paper/2505.19716/) (2025)
- [How Difficulty-Aware Staged Reinforcement Learning Enhances LLMs' Reasoning Capabilities: A Preliminary Experimental Study](https://mlsanity.com/paper/2504.00829/) (2025)
- [Train Less, Learn More: Adaptive Efficient Rollout Optimization for Group-Based Reinforcement Learning](https://mlsanity.com/paper/2602.14338/) (2026)
- [MiroMind-M1: An Open-Source Advancement in Mathematical Reasoning via Context-Aware Multi-Stage Policy Optimization](https://mlsanity.com/paper/2507.14683/) (2025)
- [CrossWordBench: Evaluating the Reasoning Capabilities of LLMs and LVLMs with Controllable Puzzle Generation](https://mlsanity.com/paper/2504.00043/) (2025)
- [StatEval: A Comprehensive Benchmark for Large Language Models in Statistics](https://mlsanity.com/paper/2510.09517/) (2025)
- [Where Did This Sentence Come From? Tracing Provenance in LLM Reasoning Distillation](https://mlsanity.com/paper/2512.20908/) (2025)
- [Timely Machine: Awareness of Time Makes Test-Time Scaling Agentic](https://mlsanity.com/paper/2601.16486/) (2026)
- [OpenReview Should be Protected and Leveraged as a Community Asset for Research in the Era of Large Language Models](https://mlsanity.com/paper/2505.21537/) (2025)
- [Weak-Driven Learning: How Weak Agents make Strong Agents Stronger](https://mlsanity.com/paper/2602.08222/) (2026)
- [Mitigating Jailbreaks with Intent-Aware LLMs](https://mlsanity.com/paper/2508.12072/) (2025)
- [The Molecular Structure of Thought: Mapping the Topology of Long Chain-of-Thought Reasoning](https://mlsanity.com/paper/2601.06002/) (2026)
- [The Challenge of Teaching Reasoning to LLMs Without RL or Distillation](https://mlsanity.com/paper/2507.09850/) (2025)
- [More Than One Teacher: Adaptive Multi-Guidance Policy Optimization for Diverse Exploration](https://mlsanity.com/paper/2510.02227/) (2025)
- [CircuitSeer: Mining High-Quality Data by Probing Mathematical Reasoning Circuits in LLMs](https://mlsanity.com/paper/2510.18470/) (2025)

## Abstract (arXiv)

The AM-DeepSeek-R1-Distilled is a large-scale dataset with thinking traces for general reasoning tasks, composed of high-quality and challenging reasoning problems. These problems are collected from a multitude of open-source datasets, subjected to semantic deduplication and meticulous cleaning to eliminate test set contamination. All responses within the dataset are distilled from reasoning models (predominantly DeepSeek-R1) and have undergone rigorous verification procedures. Mathematical problems are validated by checking against reference answers, code problems are verified using test cases, and other tasks are evaluated with the aid of a reward model. The AM-Distill-Qwen-32B model, which was trained through only simple Supervised Fine-Tuning (SFT) using this batch of data, outperformed the DeepSeek-R1-Distill-Qwen-32B model on four benchmarks: AIME2024, MATH-500, GPQA-Diamond, and LiveCodeBench. Additionally, the AM-Distill-Qwen-72B model surpassed the DeepSeek-R1-Distill-Llama-70B model on all benchmarks as well. We are releasing these 1.4 million problems and their corresponding responses to the research community with the objective of fostering the development of powerful reasoning-oriented Large Language Models (LLMs). The dataset was published in hrefhttps://huggingface.co/datasets/a-m-team/AM-DeepSeek-R1-Distilled-1.4Mhttps://huggingface.co/datasets/a-m-team/AM-DeepSeek-R1-Distilled-1.4M.