# Multi-Teacher Ensemble Distillation: A Mathematical Framework for Probability-Domain Knowledge Aggregation

arXiv: https://arxiv.org/abs/2601.09165 · HTML: https://mlsanity.com/paper/2601.09165/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in General knowledge distillation methods, one of 12 directions in Knowledge distillation.
- Builds on 8 of its 18 references that are in the map.
- First posted on arXiv 14 Jan 2026.

## Placed in

- Knowledge distillation / [General knowledge distillation methods](https://mlsanity.com/distill/general-knowledge-distillation-methods/)

## Builds on (most cited first, 8 of 8)

- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks](https://mlsanity.com/paper/1803.03635/) (2018)
- [On the Efficacy of Knowledge Distillation](https://mlsanity.com/paper/1910.01348/) (2019)
- [Adaptive Multi-Teacher Multi-level Knowledge Distillation](https://mlsanity.com/paper/2103.04062/) (2021)
- [Adaptive Multi-Teacher Knowledge Distillation with Meta-Learning](https://mlsanity.com/paper/2306.06634/) (2023)
- [Multi-teacher knowledge distillation as an effective method for compressing ensembles of neural networks](https://mlsanity.com/paper/2302.07215/) (2023)
- [Sparse Knowledge Distillation: A Mathematical Framework for Probability-Domain Temperature Scaling and Multi-Stage Compression](https://mlsanity.com/paper/2601.03195/) (2026)
- [Hallucinations Live in Variance](https://mlsanity.com/paper/2601.07058/) (2026)

## Built on by (most cited first, 3 of 3)

- [Recursive Meta-Distillation: An Axiomatic Framework for Iterative Knowledge Refinement](https://mlsanity.com/paper/2601.13100/) (2026)
- [Adaptive Weighting in Knowledge Distillation: An Axiomatic Framework for Multi-Scale Teacher Ensemble Optimization](https://mlsanity.com/paper/2601.17910/) (2026)
- [Post-Training Probability Manifold Correction via Structured SVD Pruning and Self-Referential Distillation](https://mlsanity.com/paper/2602.00372/) (2026)

## Abstract (arXiv)

Building on the probability-domain distillation framework of Sparse-KD, we develop an axiomatic, operator-theoretic framework for multi-teacher ensemble knowledge distillation. Rather than prescribing a specific aggregation formula, we define five core axioms governing valid knowledge aggregation operators, encompassing convexity, positivity, continuity, weight monotonicity, and temperature coherence. We prove the existence and non-uniqueness of operator families satisfying these axioms, establishing that multiple distinct aggregation mechanisms conform to the same foundational principles. Within this framework, we establish operator-agnostic guarantees showing that multi-teacher aggregation reduces both stochastic variance and systematic supervisory bias under heterogeneous teachers, while providing Jensen-type bounds, log-loss guarantees, and safety attenuation properties. For aggregation operators linear in teacher weights, we further establish classical ensemble variance-reduction results under standard independence assumptions, with extensions to correlated-error regimes. The framework provides theoretical grounding for multi-teacher distillation from diverse frontier models while admitting multiple valid implementation strategies.