# DTDD: Divergence-Triggered Dynamic Distillation for Reliable On-Policy Supervision

arXiv: https://arxiv.org/abs/2610.19302 · HTML: https://mlsanity.com/paper/2610.19302/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in On-policy and self-distillation for LLMs, one of 12 directions in Knowledge distillation.
- Builds on 25 of its 30 references that are in the map.
- First posted on arXiv 13 Oct 2026.

## Placed in

- Knowledge distillation / [On-policy and self-distillation for LLMs](https://mlsanity.com/distill/on-policy-and-self-distillation-for-llms/)

## Builds on (most cited first, 25 of 25)

- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [Qwen2.5 Technical Report](https://mlsanity.com/paper/2412.15115/) (2024)
- [DAPO: An Open-Source LLM Reinforcement Learning System at Scale](https://mlsanity.com/paper/2503.14476/) (2025)
- [LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code](https://mlsanity.com/paper/2403.07974/) (2024)
- [ALFWorld: Aligning Text and Embodied Environments for Interactive Learning](https://mlsanity.com/paper/2010.03768/) (2020)
- [On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes](https://mlsanity.com/paper/2306.13649/) (2023)
- [Sequence-Level Knowledge Distillation](https://mlsanity.com/paper/1606.07947/) (2016)
- [Group-in-Group Policy Optimization for LLM Agent Training](https://mlsanity.com/paper/2505.10978/) (2025)
- [OpenThoughts: Data Recipes for Reasoning Models](https://mlsanity.com/paper/2506.04178/) (2025)
- [Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipe](https://mlsanity.com/paper/2604.13016/) (2026)
- [Revisiting On-Policy Distillation: Empirical Failure Modes and Simple Fixes](https://mlsanity.com/paper/2603.25562/) (2026)
- [TCOD: Exploring Temporal Curriculum in On-Policy Distillation for Multi-turn Autonomous Agents](https://mlsanity.com/paper/2604.24005/) (2026)
- [Demystifying Reinforcement Learning in Agentic Reasoning](https://mlsanity.com/paper/2510.11701/) (2025)
- [Demystifying OPD: Length Inflation and Stabilization Strategies for Large Language Models](https://mlsanity.com/paper/2604.08527/) (2026)
- [Prune-OPD: Efficient and Reliable On-Policy Distillation for Long-Horizon Reasoning](https://mlsanity.com/paper/2605.07804/) (2026)
- [On-Policy Distillation with Curriculum Turn-level Guidance for Multi-turn Agents](https://mlsanity.com/paper/2606.15912/) (2026)
- [SOD: Step-wise On-policy Distillation for Small Language Model Agents](https://mlsanity.com/paper/2605.07725/) (2026)
- [Imitation Learning for Multi-turn LM Agents via On-policy Expert Corrections](https://mlsanity.com/paper/2512.14895/) (2025)
- [Pass the Baton: Trajectory-Relayed On-Policy Distillation](https://mlsanity.com/paper/2607.26057/) (2026)
- [Trajectory-Refined Distillation](https://mlsanity.com/paper/2606.08432/) (2026)
- [Teacher-Guided Policy Optimization for On-Policy Reasoning Distillation under Large Policy Divergence](https://mlsanity.com/paper/2605.13230/) (2026)
- [Step-Level On-Policy Distillation: Interpolating Between On-Policy Distillation and Supervised Fine-Tuning](https://mlsanity.com/paper/2608.16333/) (2026)
- [AdaSwitch: Balancing Exploration and Guidance in Knowledge Distillation via Adaptive Switching](https://mlsanity.com/paper/2510.07842/) (2025)
- [DASH-OPD: Discrepancy-Aware Switching with Hysteresis for On-Policy Distillation](https://mlsanity.com/paper/2607.29078/) (2026)

## Abstract (arXiv)

On-policy distillation (OPD) provides dense token-level supervision on student rollouts, but this supervision can become unreliable when the student and teacher diverge, either because of a large initial policy gap or compounding errors in long-horizon reasoning and tool use. We introduce Divergence-Triggered Dynamic Distillation (DTDD), an adaptive OPD framework that uses segment-level student-teacher divergence to trigger teacher takeover during training. When divergence triggers an intervention during reasoning, the student prefix continues to receive standard OPD supervision. The teacher then generates a recovery suffix, which the student learns through behavior cloning, using cross-entropy on the teacher-generated tokens. We evaluate DTDD in three regimes: long-horizon reasoning, distillation from a base initialization without offline supervised fine-tuning (SFT), and multi-turn agents. We also compare alternative trigger designs and teacher-intervention rates. DTDD reduces the post-warmup peak gradient norm by 8.6× and, from a base initialization, reaches 21.3% held-out math accuracy in 2.6× fewer optimizer updates than OPD; both methods finish near 22%. It improves best-observed performance by 4.8 and 4.1 percentage points on ALFWorld with LoRA and agentic code, respectively. Ablations show that segment-averaged triggers outperform token-spike triggers. Behavior cloning is also more effective than the tested OPD surrogate for training teacher-generated recoveries. Across 23 experimental arms, takeover tends to help when the teacher substantially outperforms the OPD-trained student, provided that the teacher can recover from student-visited states, the student can learn those recoveries, and teacher intervention is neither excessive nor insufficient.