Learn from Whoever Is Right: Answer-Verified Multi-Teacher Distillation for Multi-Domain LLMs
Not on the map. Why?
Abstract (arXiv)
Modern large language models (LLMs) rely on reinforcement learning to build strong capabilities in individual domains, but integrating those capabilities into a single deployable model remains challenging. By routing each sample to the teacher whose domain matches it, existing approaches let a domain label decide which teacher provides supervision. However, domain expertise holds only on average: the matched teacher is not always correct on a given sample, while a teacher from another domain sometimes is. The reliable teacher therefore has to be identified per sample, not per domain. In this paper, we introduce Multi-Teacher Self-Distillation Policy Optimization (MT-SDPO), an on-policy distillation method that unifies several frozen teachers into one student model. MT-SDPO consists of three components: (1) self-anchors, where a rollout is supervised by a correct rollout from its own group; (2) answer-verified eligibility, where a teacher may supervise a sample only if its own answer passes a verifier; and (3) privileged distillation, which merges the anchor and all verified feedback into one context that an exponential moving average self-teacher reads and the student does not, thereby keeping one policy at deployment. Across five students from three model families, MT-SDPO lifts the weakest domain of Qwen3-8B by 14.79 points and narrows its domain gap by 74.7%, a better balance than serving one matched teacher per domain. Verified reliability, not domain membership, should decide who teaches. Code is available at https://github.com/hexixiang/MT-SDPO.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 23 of its 29 references that are in the map. First posted on arXiv 2 Sep 2026.
Builds on
23 of 29 references.
References · 23
- 2022Training language models to follow instructions with human feedbackcited by 15,712
- 2017Proximal Policy Optimization Algorithmscited by 14,441
- 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
- 2025Qwen3 Technical Reportcited by 8,195
- 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,722
- 2025Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learningcited by 1,371
- 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,366
- 2020Gradient Surgery for Multi-Task Learningcited by 1,210
- 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Trainingcited by 988
- 2024Generative Verifiers: Reward Modeling as Next-Token Predictioncited by 465
- 2025SWE-RL: Advancing LLM Reasoning via Reinforcement Learning on Open Software Evolutioncited by 246
- 2026MiMo-V2-Flash Technical Reportcited by 163
11 more
- 2026On-Policy Context Distillation for Language Modelscited by 161
- 2025R2E-Gym: Procedural Environments and Hybrid Verifiers for Scaling Open-Weights SWE Agentscited by 140
- 2021Adaptive Multi-Teacher Multi-level Knowledge Distillationcited by 69
- 2020Reinforced Multi-Teacher Selection for Knowledge Distillationcited by 62
- 2026MOPD: Multi-Teacher On-Policy Distillation for Capability Integration in LLM Post-Trainingcited by 60
- 2026Nemotron-Cascade 2: Post-Training LLMs with Cascade RL and Multi-Domain On-Policy Distillationcited by 53
- 2026Expanding the Capabilities of Reinforcement Learning via Text Feedbackcited by 49
- 2025Nemotron-Cascade: Scaling Cascaded Reinforcement Learning for General-Purpose Reasoning Modelscited by 45
- 2026Rebellious Student: Reversing Teacher Signals for Reasoning Exploration with Self-Distilled RLVRcited by 13
- 2026Advancing General-Purpose Reasoning Models with Modular Gradient Surgerycited by 9
- 2026Disentangling Task Conflicts in Multi-Task LoRA via Orthogonal Gradient Projectioncited by 8
Built on by
Citing papers per year: 2026: 2.
2 papers.