mlsanityData to 7 Oct 2026

Beyond Teacher Assignment: Domain-Normalized Multi-Teacher On-Policy Distillation

arXiv 2609.35347 ↗First posted 28 Sep 2026cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Reinforcement learning can turn one language model into several specialists, each excellent at a single skill such as mathematics, coding or following instructions, but users need one model with all of these skills. Multi-teacher on-policy distillation (MOPD) merges them by letting the specialists teach one student: the student answers each prompt, and the specialist for that prompt's domain gives feedback on every token. This routing decides which specialist teaches, but not how strongly its feedback moves the shared student. In Qwen3.5 models at three sizes, we find that MOPD's student does not beat one taught by the best single specialist and gains little of the mathematics specialist's advantage. The feedback is unbalanced: instruction-following feedback is several times more spread out than mathematics feedback and dominates the student's updates. We propose Domain-Normalized MOPD (DN-MOPD), which keeps the routing and rescales each domain's feedback by its measured spread. On six public benchmarks, DN-MOPD improves the average score over MOPD at every size, across three random seeds and under two answer-length limits, and recovers most of the lost mathematics gain. Controls with fixed domain weights show that the gain comes mainly from turning down instruction-following feedback rather than turning up mathematics alone, and that fixed weights close to those DN-MOPD measures perform comparably. Combining specialists therefore requires deciding not only which one teaches, but also how strongly its feedback counts.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 36 of its 40 references that are in the map. First posted on arXiv 28 Sep 2026.

Builds on

36 of 40 references.

References · 36

  1. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  2. 2025Qwen3 Technical Reportcited by 8,195
  3. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  4. 2023Let's Verify Step by Stepcited by 3,598
  5. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,678
  6. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 1,906
  7. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,366
  8. 2017Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semanticscited by 1,345
  9. 2025Understanding R1-Zero-Like Training: A Critical Perspectivecited by 1,276
  10. 2023Instruction-Following Evaluation for Large Language Modelscited by 1,143
  11. 2022Editing Models with Task Arithmeticcited by 1,076
  12. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Trainingcited by 988
24 more
  1. 2026DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligencecited by 877
  2. 2023On-Policy Distillation of Language Models: Learning from Self-Generated Mistakescited by 840
  3. 2017GradNorm: Gradient Normalization for Adaptive Loss Balancing in Deep Multitask Networkscited by 816
  4. 2016Sequence-Level Knowledge Distillationcited by 761
  5. 2025Process Reinforcement through Implicit Rewardscited by 478
  6. 2025DeepMath-103K: A Large-Scale, Challenging, Decontaminated, and Verifiable Mathematical Dataset for Advancing Reasoningcited by 261
  7. 2024Advancing LLM Reasoning Generalists with Preference Treescited by 191
  8. 2025Generalizing Verifiable Instruction Followingcited by 178
  9. 2026MiMo-V2-Flash Technical Reportcited by 163
  10. 2026Learning beyond Teacher: Generalized On-Policy Distillation with Reward Extrapolationcited by 157
  11. 2024DistiLLM: Towards Streamlined Distillation for Large Language Modelscited by 145
  12. 2016Learning values across many orders of magnitudecited by 80
  13. 2026MOPD: Multi-Teacher On-Policy Distillation for Capability Integration in LLM Post-Trainingcited by 60
  14. 2026Uni-OPD: Unifying On-Policy Distillation with a Dual-Perspective Recipecited by 43
  15. 2025QoQ-Med: Building Multimodal Clinical Foundation Models with Domain-Aware GRPO Trainingcited by 38
  16. 2026DeepSeek-V4.1-Flash: Pushing the Limits of KV Cache Compressioncited by 34
  17. 2026Nemotron 3 Ultra: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoningcited by 32
  18. 2026Open-MOPD: Diagnosing and Fixing Capability Imbalance in Multi-Teacher On-Policy Distillationcited by 12
  19. 2026PowerOPD: Stabilizing On-Policy Distillation with Bounded Power Transformationcited by 7
  20. 2026UI-MOPD: Multi-Platform On-Policy Distillation for Unified GUI Agentscited by 7
  21. 2026CROP: Task Relevance via Counterfactuals for Selective On-Policy Distillationcited by 5
  22. 2026Lightning OPD 2.0: Mitigating Style Bias in Cross-Teacher On-Policy Distillation for Large Reasoning Modelscited by 3
  23. 2026Solar Open 2 Technical Reportcited by 2
  24. 2026Language-Specialized Multi-Teacher On-Policy Distillation for Multilingual LLM-Based ASRcited by 1

Built on by

Not cited yet.