# Composing What Each Teacher Learned: Multi-Teacher On-Policy Distillation through Teacher-Relative Shifts

arXiv: https://arxiv.org/abs/2610.10460 · HTML: https://mlsanity.com/paper/2610.10460/

## Key facts

- No citing papers in the map yet (data through 7 Oct 2026).
- Builds on 26 of its 34 references that are in the map.
- First posted on arXiv 7 Oct 2026.

## Builds on (most cited first, 26 of 26)

- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipe](https://mlsanity.com/paper/2604.13016/) (2026)
- [Specializing Smaller Language Models towards Multi-Step Reasoning](https://mlsanity.com/paper/2301.12726/) (2023)
- [On-Policy Context Distillation for Language Models](https://mlsanity.com/paper/2602.12275/) (2026)
- [Learning beyond Teacher: Generalized On-Policy Distillation with Reward Extrapolation](https://mlsanity.com/paper/2602.12125/) (2026)
- [A Survey of On-Policy Distillation for Large Language Models](https://mlsanity.com/paper/2604.00626/) (2026)
- [Explanations from Large Language Models Make Small Reasoners Better](https://mlsanity.com/paper/2210.06726/) (2022)
- [TIP: Token Importance in On-Policy Distillation](https://mlsanity.com/paper/2604.14084/) (2026)
- [Unifying Group-Relative and Self-Distillation Policy Optimization via Sample Routing](https://mlsanity.com/paper/2604.02288/) (2026)
- [MOPD: Multi-Teacher On-Policy Distillation for Capability Integration in LLM Post-Training](https://mlsanity.com/paper/2606.30406/) (2026)
- [Self-Distillation Zero: Self-Revision Turns Binary Rewards into Dense Supervision](https://mlsanity.com/paper/2604.12002/) (2026)
- [Black-Box On-Policy Distillation of Large Language Models](https://mlsanity.com/paper/2511.10643/) (2025)
- [CRISP: Compressed Reasoning via Iterative Self-Policy Distillation](https://mlsanity.com/paper/2603.05433/) (2026)
- [Uni-OPD: Unifying On-Policy Distillation with a Dual-Perspective Recipe](https://mlsanity.com/paper/2605.03677/) (2026)
- [Not All Disagreement Is Learnable: Token Teachability in On-Policy Distillation](https://mlsanity.com/paper/2605.26844/) (2026)
- [PACED: Distillation and On-Policy Self-Distillation at the Frontier of Student Competence](https://mlsanity.com/paper/2603.11178/) (2026)
- [Counteraction-Aware Multi-Teacher On-Policy Distillation for General Capability Recovery with Domain Preservation](https://mlsanity.com/paper/2605.27115/) (2026)
- [Weak-to-Strong Generalization via Direct On-Policy Distillation](https://mlsanity.com/paper/2607.05394/) (2026)
- [Rubric-based On-policy Distillation](https://mlsanity.com/paper/2605.07396/) (2026)
- [From Generic Correlation to Input-Specific Credit in On-Policy Self Distillation](https://mlsanity.com/paper/2605.11613/) (2026)
- [Beyond GRPO and On-Policy Distillation: An Empirical Sparse-to-Dense Reward Principle for Language-Model Post-Training](https://mlsanity.com/paper/2605.12483/) (2026)
- [On-Policy Delta Distillation](https://mlsanity.com/paper/2607.15161/) (2026)
- [UI-MOPD: Multi-Platform On-Policy Distillation for Unified GUI Agents](https://mlsanity.com/paper/2607.04425/) (2026)
- [Every Coin Has Two Sides: On the Dual Nature of Generalization in On-Policy Distillation of Large Language Models](https://mlsanity.com/paper/2608.16647/) (2026)
- [When Top-K Misses the Decision: Tool-Call Drift in Multi-Teacher On-Policy Distillation](https://mlsanity.com/paper/2607.07050/) (2026)
- [REGEN: Replay-recycling for Expert-to-Generalist distillation with Offline Reinforcement Learning](https://mlsanity.com/paper/2607.19450/) (2026)

## Abstract (arXiv)

Multi-teacher on-policy distillation (MOPD) is used in two settings. In common-domain composition, several teachers score each student rollout from one prompt domain and their signals form a single target; in routed-domain distillation, prompts from different domains are assigned to the corresponding specialist. Both settings usually transfer each teacher's endpoint policy, which mixes what post-training changed with preferences inherited from the teacher's base. We introduce Delta-MOPD, which transfers each teacher's teacher-minus-base logit shift re-anchored at the student's frozen initialization, and compare it with endpoint supervision in both settings while holding teacher selection fixed. We first expose the mechanism that impedes endpoint transfer: inherited base pull can exceed the post-training shift. Removing it reduces the teacher-term norm ratio and target--student KL. Across our experiments, the results suggest that shift targets are particularly useful when teacher signals are combined at a state. With three composed teachers, Delta-MOPD exceeds endpoint composition by 4.11 Math and 1.95 five-benchmark points; with two, it matches endpoint accuracy. Under phased routing, it achieves higher mean performance in both phase orders and reduces the observed order gap from 10.50 to 6.42 points. Under interleaved routing, where each update involves one teacher, the two targets perform comparably. The phased results provide supporting evidence that the benefit may extend to signals accumulated across training phases. Target construction is thus an independent design axis in MOPD, complementary to teacher selection.