mlsanityData to 14 Oct 2026

MOPD: Multi-Teacher On-Policy Distillation for Capability Integration in LLM Post-Training

arXiv 2606.30406 ↗First posted 29 Jun 2026cs.CL cs.LGcited by 72

On the map: Knowledge distillation → On-policy and self-distillation for LLMs

Abstract (arXiv)

Modern large language models (LLMs) rely on reinforcement learning during post-training to push specific capabilities, yet integrating multiple capabilities into one model remains hard. Existing methods, such as Off-Policy Finetune and Mix-RL, are either inefficient or lose performance. In this work, we propose Multi-teacher On-Policy Distillation (MOPD), a post-training paradigm for combining the capabilities of multiple domain RL teachers: we first run per-domain specialised RL to obtain a set of domain teachers, then distill these teachers into the student on its own rollouts. This eliminates exposure bias and provides a dense optimization signal. On Qwen3-30B-A3B, MOPD outperforms Mix-RL, Cascade RL, Off-Policy Finetune, and Param-Merge baselines, inheriting nearly all of each teacher's capability. MOPD also enables parallel, independent development of domain teachers, removing the cross-domain coupling typical of multi-domain post-training. MOPD has been deployed in the post-training of MiMo-V2-Flash, an industrial-scale frontier model, demonstrating its practical value for capability integration in frontier-scale LLMs.

Key facts. Cited by 72 papers in the map (data through 14 Oct 2026). Placed in On-policy and self-distillation for LLMs, one of 12 directions in Knowledge distillation. Builds on 22 of its 29 references that are in the map. First posted on arXiv 29 Jun 2026.

Builds on

22 of 29 references.

Closest ancestors · 1

Same direction, newest first

  1. 2026Scaling Reasoning Efficiently via Relaxed On-Policy Distillation11 Mar

Other references · 21

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  4. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  5. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,373
  6. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,745
  7. 2025Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learningcited by 1,444
  8. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,396
  9. 2025gpt-oss-120b & gpt-oss-20b Model Cardcited by 1,386
  10. 2024SimPO: Simple Preference Optimization with a Reference-Free Rewardcited by 963
  11. 2023On-Policy Distillation of Language Models: Learning from Self-Generated Mistakescited by 939
  12. 2023TIES-Merging: Resolving Interference When Merging Modelscited by 818
9 more
  1. 2016Sequence-Level Knowledge Distillationcited by 801
  2. 2023Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunchcited by 694
  3. 2015Sequence Level Training with Recurrent Neural Networkscited by 576
  4. 2025Open-Reasoner-Zero: An Open Source Approach to Scaling Up Reinforcement Learning on the Base Modelcited by 461
  5. 2019Which Tasks Should Be Learned Together in Multi-task Learning?cited by 375
  6. 2025SWE-RL: Advancing LLM Reasoning via Reinforcement Learning on Open Software Evolutioncited by 263
  7. 2025R2E-Gym: Procedural Environments and Hybrid Verifiers for Scaling Open-Weights SWE Agentscited by 158
  8. 2025Big-Math: A Large-Scale, High-Quality Math Dataset for Reinforcement Learning in Language Modelscited by 109
  9. 2024Pre-training Distillation for Large Language Models: A Design Space Explorationcited by 27

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 72.

72 papers.

Sort:

By area:

  1. 2026On-Policy Delta Distillationcited by 15
  2. 2026Does On-Policy Distillation Really Distill? From Noisy Teacher to Self-Improvementcited by 11
  3. 2026Weak-to-Strong On-Policy Distillationcited by 8
  4. 2026SenseNova-U1.5: Towards Native Unified Visual Intelligencecited by 7
  5. 2026RP-OPSD: Resolution-Privileged On-Policy Self-Distillation for Multimodal Large Language Modelscited by 6
  6. 2026What Matters in On-Policy Distillation? A Perspective on Data Efficiency and Data Selectioncited by 6
  7. 2026Full-bandwidth transformercited by 5
  8. 2026Verify Before You Distill: Prompt-Level Teacher Gating for On-Policy Distillationcited by 5
  9. 2026When Top-K Misses the Decision: Tool-Call Drift in Multi-Teacher On-Policy Distillationcited by 4
  10. 2026LLM-as-a-Coach: Experiential Learning for Non-Verifiable Taskscited by 4
  11. 2026Intern-S2-Preview: Scientific Agentic Foundation Modelcited by 4
  12. 2026Every Coin Has Two Sides: On the Dual Nature of Generalization in On-Policy Distillation of Large Language Modelscited by 4
38 more
  1. 2026Learn from Whoever Is Right: Answer-Verified Multi-Teacher Distillation for Multi-Domain LLMscited by 4
  2. 2026Solar Open 2 Technical Reportcited by 3
  3. 2026Outcome-Confounded Local Supervision in On-Policy Distillationcited by 3
  4. 2026Who Teaches Which Token? Verifier-Gated Multi-Expert On-Policy Distillation for Scientific Reasoningcited by 3
  5. 2026REGEN: Replay-recycling for Expert-to-Generalist distillation with Offline Reinforcement Learningcited by 2
  6. 2026D3-MOPD: Dynamic Domain ScheDuling for Efficient Multi-Teacher Distillationcited by 2
  7. 2026Eliciting Weak-to-Strong Generalization with On-Policy Reverse Distillationcited by 2
  8. 2026Distill What You Trust: Reliability-Aware Multi-Teacher On-Policy Distillationcited by 2
  9. 2026Qwen3.8-Omni: Towards Native Omni-Modal Agentscited by 2
  10. 2026OPOD: On-Policy Omni Distillationcited by 1
  11. 2026FlowCTS: On-policy Continuous Trajectory Supervision of Flow Modelscited by 1
  12. 2026Beyond the Best Teacher: Expanding and Compressing the Reasoning Solution Manifoldcited by 1
  13. 2026Language-Specialized Multi-Teacher On-Policy Distillation for Multilingual LLM-Based ASRcited by 1
  14. 2026Bidirectional Context Self-Distillation for Reinforcement Learning of Skill-Based LLM Agentscited by 1
  15. 2026Scheduling Mixed RL Rollouts Beyond Prefix Localitycited by 1
  16. 2026Data-Free On-Policy Distillation: How Far Can We Go Without External Data?cited by 1
  17. 2026ME-VLM: A Unified VLM for Embodied Cognition and Agent Coordinationcited by 1
  18. 2026MMVistaReason: Toward Open-Data and Post-Training Recipes for Multimodal Reasoningcited by 1
  19. 2026MiMo-V2.6: Scaling Reinforcement Learning Towards Self-Improvementcited by 1
  20. 2026UMoE:Unlocking Every Expert in Domain-Specific Trainingcited by 0
  21. 2026Capek 0.5: An Execution-Centric Vision-Language Model for Embodied Intelligencecited by 0
  22. 2026From Passive Delegates to Strategic Negotiators: Reinforcing Social Reasoning in Small Language Models with SocialRLcited by 0
  23. 2026Mint-Agent: Introducing Finance-Native Agentic Foundation Modelscited by 0
  24. 2026One Symptom, Three Levers: A Critical Review of On-Policy Self-Distillationcited by 0
  25. 2026Instella-MoE Technical Reportcited by 0
  26. 2026Feyospace-v1: How the Cyber Mercury Seven Trained Frontier Cyber Modelscited by 0
  27. 2026CompassOPD: Cross-Family On-Policy Distillation via Within-Family Likelihood Shiftscited by 0
  28. 2026Lightning Weave: Improving the Accuracy-Efficiency Frontier of Reasoning Models through Capability Compositioncited by 0
  29. 2026UniPolicy: Unified Objective-Specific Policies for Generative Search Advertisingcited by 0
  30. 2026A Shared Learning Rate Is Not a Neutral Control in Selective On-Policy Distillationcited by 0
  31. 2026MOPD-Router: Rethinking Teacher Routing in Multi-Teacher On-Policy Distillationcited by 0
  32. 2026DuoOPD: Learning from Joint Teacher-Student Outcomes for Multi-Task On-Policy Distillationcited by 0
  33. 2026Do We Really Need KL Divergence for On-Policy Distillation of Large Language Models?cited by 0
  34. 2026ChemOPD: Multi-Teacher On-Policy Distillation for Multi-Task Chemical Reasoningcited by 0
  35. 2026USA: Update-aware SAM for Cross-domain On-Policy Disitllation of Language Agentscited by 0
  36. 2026PMOPD: Task Ordering, Cycling, and Parameter-Update Subspace Protection in Multi-Teacher On-Policy Distillationcited by 0
  37. 2026No Pain, More Gain: Iterative Merging for Effective Multi-Teacher On-Policy Distillationcited by 0
  38. 2026DivOPD: Spread Wide, Look Close for Asynchronous On-Policy Distillation of Multi-turn Agentscited by 0

50 most cited of 72 All 72 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Open-MOPD: Diagnosing and Fixing Capability Imbalance in Multi-Teacher On-Policy Distillation
  2. 2026Counteraction-Aware Multi-Teacher On-Policy Distillation for General Capability Recovery with Domain Preservation
  3. 2026Distilled Reinforcement Learning for LLM Post-training
  4. 2026Recovering General Capabilities via Uncertainty-Calibrated Multi-Teacher On-Policy Distillation
  5. 2026Teacher-Guided Policy Optimization for On-Policy Reasoning Distillation under Large Policy Divergence
  6. 2026Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipe
  7. 2026Large Language Model Post-Training: A Unified View of Off-Policy and On-Policy Learning
  8. 2026Be My Tutor: On-Policy Co-Distillation for Mutual LLM Improvement via Peer Feedback