# MOPD: Multi-Teacher On-Policy Distillation for Capability Integration in LLM Post-Training

arXiv: https://arxiv.org/abs/2606.30406 · HTML: https://mlsanity.com/paper/2606.30406/

## Key facts

- Cited by 72 papers in the map (data through 14 Oct 2026).
- Placed in On-policy and self-distillation for LLMs, one of 12 directions in Knowledge distillation.
- Builds on 22 of its 29 references that are in the map.
- First posted on arXiv 29 Jun 2026.

## Placed in

- Knowledge distillation / [On-policy and self-distillation for LLMs](https://mlsanity.com/distill/on-policy-and-self-distillation-for-llms/)

## Builds on (most cited first, 22 of 22)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [SWE-bench: Can Language Models Resolve Real-World GitHub Issues?](https://mlsanity.com/paper/2310.06770/) (2023)
- [WebGPT: Browser-assisted question-answering with human feedback](https://mlsanity.com/paper/2112.09332/) (2021)
- [Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning](https://mlsanity.com/paper/2503.09516/) (2025)
- [Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time](https://mlsanity.com/paper/2203.05482/) (2022)
- [gpt-oss-120b & gpt-oss-20b Model Card](https://mlsanity.com/paper/2508.10925/) (2025)
- [SimPO: Simple Preference Optimization with a Reference-Free Reward](https://mlsanity.com/paper/2405.14734/) (2024)
- [On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes](https://mlsanity.com/paper/2306.13649/) (2023)
- [TIES-Merging: Resolving Interference When Merging Models](https://mlsanity.com/paper/2306.01708/) (2023)
- [Sequence-Level Knowledge Distillation](https://mlsanity.com/paper/1606.07947/) (2016)
- [Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunch](https://mlsanity.com/paper/2311.03099/) (2023)
- [Sequence Level Training with Recurrent Neural Networks](https://mlsanity.com/paper/1511.06732/) (2015)
- [Open-Reasoner-Zero: An Open Source Approach to Scaling Up Reinforcement Learning on the Base Model](https://mlsanity.com/paper/2503.24290/) (2025)
- [Which Tasks Should Be Learned Together in Multi-task Learning?](https://mlsanity.com/paper/1905.07553/) (2019)
- [SWE-RL: Advancing LLM Reasoning via Reinforcement Learning on Open Software Evolution](https://mlsanity.com/paper/2502.18449/) (2025)
- [R2E-Gym: Procedural Environments and Hybrid Verifiers for Scaling Open-Weights SWE Agents](https://mlsanity.com/paper/2504.07164/) (2025)
- [Big-Math: A Large-Scale, High-Quality Math Dataset for Reinforcement Learning in Language Models](https://mlsanity.com/paper/2502.17387/) (2025)
- [Scaling Reasoning Efficiently via Relaxed On-Policy Distillation](https://mlsanity.com/paper/2603.11137/) (2026)
- [Pre-training Distillation for Large Language Models: A Design Space Exploration](https://mlsanity.com/paper/2410.16215/) (2024)

## Built on by (most cited first, 30 of 72)

- [On-Policy Delta Distillation](https://mlsanity.com/paper/2607.15161/) (2026)
- [Does On-Policy Distillation Really Distill? From Noisy Teacher to Self-Improvement](https://mlsanity.com/paper/2608.31046/) (2026)
- [Weak-to-Strong On-Policy Distillation](https://mlsanity.com/paper/2607.26246/) (2026)
- [SenseNova-U1.5: Towards Native Unified Visual Intelligence](https://mlsanity.com/paper/2609.11929/) (2026)
- [RP-OPSD: Resolution-Privileged On-Policy Self-Distillation for Multimodal Large Language Models](https://mlsanity.com/paper/2607.24447/) (2026)
- [What Matters in On-Policy Distillation? A Perspective on Data Efficiency and Data Selection](https://mlsanity.com/paper/2609.05198/) (2026)
- [Full-bandwidth transformer](https://mlsanity.com/paper/2608.08888/) (2026)
- [Verify Before You Distill: Prompt-Level Teacher Gating for On-Policy Distillation](https://mlsanity.com/paper/2609.02998/) (2026)
- [When Top-K Misses the Decision: Tool-Call Drift in Multi-Teacher On-Policy Distillation](https://mlsanity.com/paper/2607.07050/) (2026)
- [LLM-as-a-Coach: Experiential Learning for Non-Verifiable Tasks](https://mlsanity.com/paper/2607.18110/) (2026)
- [Intern-S2-Preview: Scientific Agentic Foundation Model](https://mlsanity.com/paper/2608.13505/) (2026)
- [Every Coin Has Two Sides: On the Dual Nature of Generalization in On-Policy Distillation of Large Language Models](https://mlsanity.com/paper/2608.16647/) (2026)
- [Learn from Whoever Is Right: Answer-Verified Multi-Teacher Distillation for Multi-Domain LLMs](https://mlsanity.com/paper/2609.02548/) (2026)
- [Solar Open 2 Technical Report](https://mlsanity.com/paper/2607.20062/) (2026)
- [Outcome-Confounded Local Supervision in On-Policy Distillation](https://mlsanity.com/paper/2607.23731/) (2026)
- [Who Teaches Which Token? Verifier-Gated Multi-Expert On-Policy Distillation for Scientific Reasoning](https://mlsanity.com/paper/2609.15404/) (2026)
- [REGEN: Replay-recycling for Expert-to-Generalist distillation with Offline Reinforcement Learning](https://mlsanity.com/paper/2607.19450/) (2026)
- [D3-MOPD: Dynamic Domain ScheDuling for Efficient Multi-Teacher Distillation](https://mlsanity.com/paper/2608.24987/) (2026)
- [Eliciting Weak-to-Strong Generalization with On-Policy Reverse Distillation](https://mlsanity.com/paper/2609.08798/) (2026)
- [Distill What You Trust: Reliability-Aware Multi-Teacher On-Policy Distillation](https://mlsanity.com/paper/2609.23697/) (2026)
- [Qwen3.8-Omni: Towards Native Omni-Modal Agents](https://mlsanity.com/paper/2609.25611/) (2026)
- [OPOD: On-Policy Omni Distillation](https://mlsanity.com/paper/2607.20918/) (2026)
- [FlowCTS: On-policy Continuous Trajectory Supervision of Flow Models](https://mlsanity.com/paper/2607.24522/) (2026)
- [Beyond the Best Teacher: Expanding and Compressing the Reasoning Solution Manifold](https://mlsanity.com/paper/2607.27770/) (2026)
- [Language-Specialized Multi-Teacher On-Policy Distillation for Multilingual LLM-Based ASR](https://mlsanity.com/paper/2608.03610/) (2026)
- [Bidirectional Context Self-Distillation for Reinforcement Learning of Skill-Based LLM Agents](https://mlsanity.com/paper/2608.09555/) (2026)
- [Scheduling Mixed RL Rollouts Beyond Prefix Locality](https://mlsanity.com/paper/2608.11152/) (2026)
- [Data-Free On-Policy Distillation: How Far Can We Go Without External Data?](https://mlsanity.com/paper/2609.14193/) (2026)
- [ME-VLM: A Unified VLM for Embodied Cognition and Agent Coordination](https://mlsanity.com/paper/2609.24526/) (2026)
- [MMVistaReason: Toward Open-Data and Post-Training Recipes for Multimodal Reasoning](https://mlsanity.com/paper/2610.01352/) (2026)

## Abstract (arXiv)

Modern large language models (LLMs) rely on reinforcement learning during post-training to push specific capabilities, yet integrating multiple capabilities into one model remains hard. Existing methods, such as Off-Policy Finetune and Mix-RL, are either inefficient or lose performance. In this work, we propose Multi-teacher On-Policy Distillation (MOPD), a post-training paradigm for combining the capabilities of multiple domain RL teachers: we first run per-domain specialised RL to obtain a set of domain teachers, then distill these teachers into the student on its own rollouts. This eliminates exposure bias and provides a dense optimization signal. On Qwen3-30B-A3B, MOPD outperforms Mix-RL, Cascade RL, Off-Policy Finetune, and Param-Merge baselines, inheriting nearly all of each teacher's capability. MOPD also enables parallel, independent development of domain teachers, removing the cross-domain coupling typical of multi-domain post-training. MOPD has been deployed in the post-training of MiMo-V2-Flash, an industrial-scale frontier model, demonstrating its practical value for capability integration in frontier-scale LLMs.