# What Matters in On-Policy Distillation? A Perspective on Data Efficiency and Data Selection

arXiv: https://arxiv.org/abs/2609.05198 · HTML: https://mlsanity.com/paper/2609.05198/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Placed in Prompting and chain-of-thought reasoning, one of 17 directions in LLM reasoning.
- Builds on 26 of its 29 references that are in the map.
- First posted on arXiv 4 Sep 2026.

## Placed in

- LLM reasoning / [Prompting and chain-of-thought reasoning](https://mlsanity.com/lr/prompting-and-chain-of-thought-reasoning/)
- Knowledge distillation / [On-policy and self-distillation for LLMs](https://mlsanity.com/distill/on-policy-and-self-distillation-for-llms/)

## Builds on (most cited first, 26 of 26)

- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [Measuring Mathematical Problem Solving With the MATH Dataset](https://mlsanity.com/paper/2103.03874/) (2021)
- [DAPO: An Open-Source LLM Reinforcement Learning System at Scale](https://mlsanity.com/paper/2503.14476/) (2025)
- [HybridFlow: A Flexible and Efficient RLHF Framework](https://mlsanity.com/paper/2409.19256/) (2024)
- [LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code](https://mlsanity.com/paper/2403.07974/) (2024)
- [Solving Quantitative Reasoning Problems with Language Models](https://mlsanity.com/paper/2206.14858/) (2022)
- [OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problems](https://mlsanity.com/paper/2402.14008/) (2024)
- [DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence](https://mlsanity.com/paper/2606.19348/) (2026)
- [On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes](https://mlsanity.com/paper/2306.13649/) (2023)
- [GLM-5: from Vibe Coding to Agentic Engineering](https://mlsanity.com/paper/2602.15763/) (2026)
- [LESS: Selecting Influential Data for Targeted Instruction Tuning](https://mlsanity.com/paper/2402.04333/) (2024)
- [Policy Distillation](https://mlsanity.com/paper/1511.06295/) (2015)
- [Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipe](https://mlsanity.com/paper/2604.13016/) (2026)
- [Reinforcement Learning for Reasoning in Large Language Models with One Training Example](https://mlsanity.com/paper/2504.20571/) (2025)
- [MiMo-V2-Flash Technical Report](https://mlsanity.com/paper/2601.02780/) (2026)
- [Entropy-Aware On-Policy Distillation of Language Models](https://mlsanity.com/paper/2603.07079/) (2026)
- [Scaling Reasoning Efficiently via Relaxed On-Policy Distillation](https://mlsanity.com/paper/2603.11137/) (2026)
- [MOPD: Multi-Teacher On-Policy Distillation for Capability Integration in LLM Post-Training](https://mlsanity.com/paper/2606.30406/) (2026)
- [JustRL: Scaling a 1.5B LLM with a Simple RL Recipe](https://mlsanity.com/paper/2512.16649/) (2025)
- [Not All Disagreement Is Learnable: Token Teachability in On-Policy Distillation](https://mlsanity.com/paper/2605.26844/) (2026)
- [Large-Scale Data Selection for Instruction Tuning](https://mlsanity.com/paper/2503.01807/) (2025)
- [Filter, Then Reweight: Rethinking Optimization Granularity in On-Policy Distillation](https://mlsanity.com/paper/2606.02684/) (2026)
- [Prune-OPD: Efficient and Reliable On-Policy Distillation for Long-Horizon Reasoning](https://mlsanity.com/paper/2605.07804/) (2026)
- [KL for a KL: On-Policy Distillation with Control Variate Baseline](https://mlsanity.com/paper/2605.07865/) (2026)
- [Rethinking On-Policy Distillation of Large Language Models II: One Training Example](https://mlsanity.com/paper/2609.04172/) (2026)
- [Single-Rollout Hidden-State Dynamics for Training-Free RLVR Data Selection](https://mlsanity.com/paper/2605.28631/) (2026)

## Built on by (most cited first, 6 of 6)

- [Ecdysis: Efficient and Effective Training of Runtime Harnesses for LLM Agents](https://mlsanity.com/paper/2609.11677/) (2026)
- [Data-Free On-Policy Distillation: How Far Can We Go Without External Data?](https://mlsanity.com/paper/2609.14193/) (2026)
- [Trapped by Their Own Rollouts: Understanding Aggregation--Rollout Feedback in Federated On-Policy Distillation](https://mlsanity.com/paper/2609.32573/) (2026)
- [Rethinking Self-Distillation for Multi-Teacher Capability Merging](https://mlsanity.com/paper/2610.04272/) (2026)
- [Learning More by Training Less: Uncertainty-Guided Progressive Contraction for Efficient Reasoning Post-Training](https://mlsanity.com/paper/2610.12933/) (2026)
- [MedEvo: Medical Reasoning Evolution through Knowledge Navigation](https://mlsanity.com/paper/2610.17690/) (2026)

## Abstract (arXiv)

On-Policy Distillation (OPD) has emerged as a widely adopted post-training paradigm for enhancing large language models in reasoning domains. However, the data-centric mechanisms in OPD remain relatively underexplored. This paper presents an empirical study of data efficiency and data selection in OPD. We begin by investigating an extreme setting: training OPD on only one example, namely 1-shot OPD. Surprisingly, we find that 1-shot OPD is consistently effective across all sampled training examples and harder examples often yield superior performance gain. We next investigate what actually drives the student model's improvement in the training data. Our analysis reveals that the improvement is not driven by high token entropy, but the longer CoT paths which hard problem naturally generate. Training on longer CoT can help maintain closer alignment with the teacher over a long reasoning horizon, and learn critical thinking patterns usually missing in short CoTs, such as reflection (e.g., "Alternatively"). Based on these insights, we propose a simple data selection method that selects only hard examples for training, where even "unsolvable" examples that completely exceed the teacher's capability can be successfully used. Our experiments conducted on five models ranging from 1.5B to 8B show that training the student model on only 8 selected hard examples matches the performance of the 17K dataset baseline. Code and models will be publicly available.