# Recursive Meta-Distillation: An Axiomatic Framework for Iterative Knowledge Refinement

arXiv: https://arxiv.org/abs/2601.13100 · HTML: https://mlsanity.com/paper/2601.13100/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in General knowledge distillation methods, one of 12 directions in Knowledge distillation.
- Builds on 8 of its 15 references that are in the map.
- First posted on arXiv 19 Jan 2026.

## Placed in

- Knowledge distillation / [General knowledge distillation methods](https://mlsanity.com/distill/general-knowledge-distillation-methods/)

## Builds on (most cited first, 8 of 8)

- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Born Again Neural Networks](https://mlsanity.com/paper/1805.04770/) (2018)
- [A Comprehensive Survey on Knowledge Distillation](https://mlsanity.com/paper/2503.12067/) (2025)
- [Smaller but Better: Self-Paced Knowledge Distillation for Lightweight yet Effective LCMs](https://mlsanity.com/paper/2408.03680/) (2024)
- [Sparse Knowledge Distillation: A Mathematical Framework for Probability-Domain Temperature Scaling and Multi-Stage Compression](https://mlsanity.com/paper/2601.03195/) (2026)
- [Hallucinations Live in Variance](https://mlsanity.com/paper/2601.07058/) (2026)
- [Multi-Teacher Ensemble Distillation: A Mathematical Framework for Probability-Domain Knowledge Aggregation](https://mlsanity.com/paper/2601.09165/) (2026)
- [UNDO: Understanding Distillation as Optimization](https://mlsanity.com/paper/2504.02521/) (2025)

## Built on by (most cited first, 2 of 2)

- [Adaptive Weighting in Knowledge Distillation: An Axiomatic Framework for Multi-Scale Teacher Ensemble Optimization](https://mlsanity.com/paper/2601.17910/) (2026)
- [Post-Training Probability Manifold Correction via Structured SVD Pruning and Self-Referential Distillation](https://mlsanity.com/paper/2602.00372/) (2026)

## Abstract (arXiv)

Recent work in probability-domain knowledge distillation has established axiomatic frameworks for temperature scaling, multi-teacher aggregation, and bias-variance trade-offs in single-stage settings. However, the mathematical behavior of recursive or multi-generation distillation remains poorly understood, with prior approaches relying primarily on empirical heuristics. In this work, we introduce an axiomatic and operator-theoretic framework for recursive meta-distillation, formalizing iterative knowledge distillation as a sequence of probability-distribution operators with explicit anchoring to base teachers. We define structural axioms for valid meta-teacher construction and prove the existence of non-trivial operator families satisfying these axioms without specifying particular algorithms or loss functions. Under mild realizability and convexity assumptions, we show that anchored recursive distillation induces contraction in KL divergence, yielding geometric convergence to base teacher distributions and a unique, globally attractive fixed point. The contribution is foundational rather than algorithmic: the framework characterizes when recursive distillation is mathematically well-posed and convergent rather than error-accumulating, independent of model architecture, optimization details, or specific operator instantiations. These results provide a theoretical basis for understanding stability, bias-variance behavior, and failure modes in iterative and multi-teacher distillation under capacity constraints.