# No Pain, More Gain: Iterative Merging for Effective Multi-Teacher On-Policy Distillation

arXiv: https://arxiv.org/abs/2609.34745 · HTML: https://mlsanity.com/paper/2609.34745/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in On-policy and self-distillation for LLMs, one of 12 directions in Knowledge distillation.
- Builds on 28 of its 29 references that are in the map.
- First posted on arXiv 28 Sep 2026.

## Placed in

- Knowledge distillation / [On-policy and self-distillation for LLMs](https://mlsanity.com/distill/on-policy-and-self-distillation-for-llms/)

## Builds on (most cited first, 28 of 28)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time](https://mlsanity.com/paper/2203.05482/) (2022)
- [What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Exams](https://mlsanity.com/paper/2009.13081/) (2020)
- [Averaging Weights Leads to Wider Optima and Better Generalization](https://mlsanity.com/paper/1803.05407/) (2018)
- [Editing Models with Task Arithmetic](https://mlsanity.com/paper/2212.04089/) (2022)
- [DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence](https://mlsanity.com/paper/2606.19348/) (2026)
- [On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes](https://mlsanity.com/paper/2306.13649/) (2023)
- [Sequence-Level Knowledge Distillation](https://mlsanity.com/paper/1606.07947/) (2016)
- [SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-training](https://mlsanity.com/paper/2501.17161/) (2025)
- [τ2-Bench: Evaluating Conversational Agents in a Dual-Control Environment](https://mlsanity.com/paper/2506.07982/) (2025)
- [FinQA: A Dataset of Numerical Reasoning over Financial Data](https://mlsanity.com/paper/2109.00122/) (2021)
- [OpenThoughts: Data Recipes for Reasoning Models](https://mlsanity.com/paper/2506.04178/) (2025)
- [Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipe](https://mlsanity.com/paper/2604.13016/) (2026)
- [When Scaling Meets LLM Finetuning: The Effect of Data, Model and Finetuning Method](https://mlsanity.com/paper/2402.17193/) (2024)
- [When Does Pretraining Help? Assessing Self-Supervised Learning for Law and the CaseHOLD Dataset](https://mlsanity.com/paper/2104.08671/) (2021)
- [Generalizing Verifiable Instruction Following](https://mlsanity.com/paper/2507.02833/) (2025)
- [Kimi K3: Open Frontier Intelligence](https://mlsanity.com/paper/2607.24653/) (2026)
- [Nemotron 3 Nano: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoning](https://mlsanity.com/paper/2512.20848/) (2025)
- [Speculative Knowledge Distillation: Bridging the Teacher-Student Gap Through Interleaved Sampling](https://mlsanity.com/paper/2410.11325/) (2024)
- [MOPD: Multi-Teacher On-Policy Distillation for Capability Integration in LLM Post-Training](https://mlsanity.com/paper/2606.30406/) (2026)
- [Nemotron-Cascade 2: Post-Training LLMs with Cascade RL and Multi-Domain On-Policy Distillation](https://mlsanity.com/paper/2603.19220/) (2026)
- [Nemotron-Cascade: Scaling Cascaded Reinforcement Learning for General-Purpose Reasoning Models](https://mlsanity.com/paper/2512.13607/) (2025)
- [The Many Faces of On-Policy Distillation: Pitfalls, Mechanisms, and Fixes](https://mlsanity.com/paper/2605.11182/) (2026)
- [Nemotron 3 Ultra: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoning](https://mlsanity.com/paper/2606.15007/) (2026)
- [Trust Region On-Policy Distillation](https://mlsanity.com/paper/2606.01249/) (2026)
- [Introducing LongCat-Flash-Thinking: A Technical Report](https://mlsanity.com/paper/2509.18883/) (2025)

## Abstract (arXiv)

Multi-teacher on-policy distillation (MOPD) combines independently developed domain teachers into a single student by distilling their predictions on student-generated samples. We study a setting where teachers share a reference model but undergo different post-training procedures, and find that MOPD can struggle to recover some teacher capabilities. Because distillation occurs on student-generated prefixes, the student initialization can strongly affect subsequent recovery. However, initial benchmark performance is not a reliable predictor of a good MOPD initialization. For example, merge initialization can start below SFT warm-up yet finish higher after MOPD. We further find that effective merging depends on both the relative teacher contributions and the overall merge scale, with some strong configurations lying outside the simplex of convex parameter averaging. Thus, selecting a good merge initialization requires evaluating not only its immediate performance but also the learning it enables under MOPD, making one-shot coefficient search difficult. We propose Iterative Merging for MOPD (IM-MOPD), which starts from a uniform merge and progressively adds task-vector increments for under-recovered domains during distillation. In a 5-domain setting, IM-MOPD achieves higher average normalized recovery than MOPD with either uniform merge initialization or SFT warm-up, showing that effective teacher contributions can be determined progressively during training.