# Open-MOPD: Diagnosing and Fixing Capability Imbalance in Multi-Teacher On-Policy Distillation arXiv: https://arxiv.org/abs/2608.19098 ยท HTML: https://mlsanity.com/paper/2608.19098/ ## Key facts - Cited by 18 papers in the map (data through 14 Oct 2026). - Placed in Prompting and chain-of-thought reasoning, one of 17 directions in LLM reasoning. - Builds on 23 of its 36 references that are in the map. - First posted on arXiv 19 Aug 2026. ## Placed in - LLM reasoning / [Prompting and chain-of-thought reasoning](https://mlsanity.com/lr/prompting-and-chain-of-thought-reasoning/) - Knowledge distillation / [On-policy and self-distillation for LLMs](https://mlsanity.com/distill/on-policy-and-self-distillation-for-llms/) ## Builds on (most cited first, 23 of 23) - [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017) - [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015) - [Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks](https://mlsanity.com/paper/1506.03099/) (2015) - [Sequence-Level Knowledge Distillation](https://mlsanity.com/paper/1606.07947/) (2016) - [Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunch](https://mlsanity.com/paper/2311.03099/) (2023) - [Sequence Level Training with Recurrent Neural Networks](https://mlsanity.com/paper/1511.06732/) (2015) - [Improved Knowledge Distillation via Teacher Assistant](https://mlsanity.com/paper/1902.03393/) (2019) - [On the Efficacy of Knowledge Distillation](https://mlsanity.com/paper/1910.01348/) (2019) - [Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipe](https://mlsanity.com/paper/2604.13016/) (2026) - [AReaL: A Large-Scale Asynchronous Reinforcement Learning System for Language Reasoning](https://mlsanity.com/paper/2505.24298/) (2025) - [Entropy-Aware On-Policy Distillation of Language Models](https://mlsanity.com/paper/2603.07079/) (2026) - [A Survey of On-Policy Distillation for Large Language Models](https://mlsanity.com/paper/2604.00626/) (2026) - [R2E-Gym: Procedural Environments and Hybrid Verifiers for Scaling Open-Weights SWE Agents](https://mlsanity.com/paper/2504.07164/) (2025) - [Branch-Train-MiX: Mixing Expert LLMs into a Mixture-of-Experts LLM](https://mlsanity.com/paper/2403.07816/) (2024) - [Speculative Knowledge Distillation: Bridging the Teacher-Student Gap Through Interleaved Sampling](https://mlsanity.com/paper/2410.11325/) (2024) - [Prosperity before Collapse: How Far Can Off-Policy RL Reach with Stale Data on LLMs?](https://mlsanity.com/paper/2510.01161/) (2025) - [Nemotron-Cascade: Scaling Cascaded Reinforcement Learning for General-Purpose Reasoning Models](https://mlsanity.com/paper/2512.13607/) (2025) - [RollPacker: Mitigating Long-Tail Rollouts for Fast, Synchronous RL Post-Training](https://mlsanity.com/paper/2509.21009/) (2025) - [MoDoMoDo: Multi-Domain Data Mixtures for Multimodal LLM Reinforcement Learning](https://mlsanity.com/paper/2505.24871/) (2025) - [Scaling the Horizon, Not the Parameters: Reaching Trillion-Parameter Performance with a 35B Agent](https://mlsanity.com/paper/2606.30616/) (2026) - [Imbalanced Gradients in RL Post-Training of Multi-Task LLMs](https://mlsanity.com/paper/2510.19178/) (2025) - [A-3PO: Accelerating Asynchronous LLM Training with Staleness-aware Proximal Policy Approximation](https://mlsanity.com/paper/2512.06547/) (2025) - [BTS: Harmonizing Specialized Experts into a Generalist LLM](https://mlsanity.com/paper/2502.00075/) (2025) ## Built on by (most cited first, 18 of 18) - [Distill What You Trust: Reliability-Aware Multi-Teacher On-Policy Distillation](https://mlsanity.com/paper/2609.23697/) (2026) - [Data-Free On-Policy Distillation: How Far Can We Go Without External Data?](https://mlsanity.com/paper/2609.14193/) (2026) - [MMVistaReason: Toward Open-Data and Post-Training Recipes for Multimodal Reasoning](https://mlsanity.com/paper/2610.01352/) (2026) - [Lightning Weave: Improving the Accuracy-Efficiency Frontier of Reasoning Models through Capability Composition](https://mlsanity.com/paper/2609.14708/) (2026) - [One to More, More to One: Category-Aware Iterative Expert Training for Software Engineering Agents](https://mlsanity.com/paper/2609.23377/) (2026) - [MOPD-Router: Rethinking Teacher Routing in Multi-Teacher On-Policy Distillation](https://mlsanity.com/paper/2609.30837/) (2026) - [Do We Really Need KL Divergence for On-Policy Distillation of Large Language Models?](https://mlsanity.com/paper/2609.33791/) (2026) - [PMOPD: Task Ordering, Cycling, and Parameter-Update Subspace Protection in Multi-Teacher On-Policy Distillation](https://mlsanity.com/paper/2609.34605/) (2026) - [DivOPD: Spread Wide, Look Close for Asynchronous On-Policy Distillation of Multi-turn Agents](https://mlsanity.com/paper/2609.34838/) (2026) - [Beyond Teacher Assignment: Domain-Normalized Multi-Teacher On-Policy Distillation](https://mlsanity.com/paper/2609.35347/) (2026) - [No Task Vector Is an Island: A Comprehensive Study on the Composability of Task Vectors from On-Policy Distillation](https://mlsanity.com/paper/2609.39405/) (2026) - [From Gradients to Capabilities: Understanding Multi-Teacher On-Policy Distillation](https://mlsanity.com/paper/2610.02179/) (2026) - [Slow-Fast Multi-Teacher On-Policy Distillation for Capability Preservation](https://mlsanity.com/paper/2610.02324/) (2026) - [Latent-MOPD: Latent Multi-Teacher On-Policy Distillation](https://mlsanity.com/paper/2610.02381/) (2026) - [Rethinking Self-Distillation for Multi-Teacher Capability Merging](https://mlsanity.com/paper/2610.04272/) (2026) - [UP-MOPD: Update Projection in Multi-Teacher On-Policy Distillation](https://mlsanity.com/paper/2610.08398/) (2026) - [Beyond Domain-Routed Supervision: Cross-Teacher Guarding for Multi-Teacher On-Policy Distillation](https://mlsanity.com/paper/2610.13521/) (2026) - [RM-OPD: On-Policy Distillation for Generative Reward Models](https://mlsanity.com/paper/2610.20174/) (2026) ## Abstract (arXiv) Multi-teacher on-policy distillation (M-OPD) has emerged as a promising paradigm for consolidating domain-specialized reinforcement learning (RL) experts into a single generalist student via dense, token-level reward supervision. Despite its practical success, the optimization dynamics governing multi-teacher capability integration remain poorly understood, and open, rigorously reproducible recipes are conspicuously lacking. In this work, we establish a controlled M-OPD benchmark on SmolLM3-3B-Base with oracle routing, isolating capability integration from routing ambiguity. Our investigation reveals a pronounced capability integration gap: standard M-OPD captures only 35.6% of the available headroom relative to a domain-routed oracle ensemble, with concise tasks such as instruction following suffering severe degradation and premature stagnation. Crucially, we show that this failure stems not from gradient conflict, but from a severe misallocation of the token-level optimization budget. This pathology is driven by three orthogonal factors: structural sequence-length disparities across domains, dynamic convergence drift due to non-uniform learning rates, and multi-step reward staleness from asynchronous policy updates. To resolve these imbalances, we introduce Open-MOPD, a principled framework incorporating token-share balancing, gap-aware dynamic budget allocation, and student reward refresh. Together, these mechanisms systematically restore cross-domain balance, elevating headroom recovery from 35.6% to 83.4% in a single deployable student. We fully open-source our end-to-end post-training recipe, training trajectories, and evaluation suites on an academically accessible hardware budget.