# Beyond Dark Knowledge: Mixup-Based Distillation for Reliable Predictions

arXiv: https://arxiv.org/abs/2606.12171 · HTML: https://mlsanity.com/paper/2606.12171/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in General knowledge distillation methods, one of 12 directions in Knowledge distillation.
- Builds on 29 of its 41 references that are in the map.
- First posted on arXiv 10 Jun 2026.

## Placed in

- Knowledge distillation / [General knowledge distillation methods](https://mlsanity.com/distill/general-knowledge-distillation-methods/)

## Builds on (most cited first, 29 of 29)

- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [Benchmarking Neural Network Robustness to Common Corruptions and Perturbations](https://mlsanity.com/paper/1903.12261/) (2019)
- [FitNets: Hints for Thin Deep Nets](https://mlsanity.com/paper/1412.6550/) (2014)
- [Knowledge Distillation: A Survey](https://mlsanity.com/paper/2006.05525/) (2020)
- [MixMatch: A Holistic Approach to Semi-Supervised Learning](https://mlsanity.com/paper/1905.02249/) (2019)
- [When Does Label Smoothing Help?](https://mlsanity.com/paper/1906.02629/) (2019)
- [Relational Knowledge Distillation](https://mlsanity.com/paper/1904.05068/) (2019)
- [Contrastive Representation Distillation](https://mlsanity.com/paper/1910.10699/) (2019)
- [Variational Information Distillation for Knowledge Transfer](https://mlsanity.com/paper/1904.05835/) (2019)
- [Knowledge Distillation and Student-Teacher Learning for Visual Intelligence: A Review and New Outlooks](https://mlsanity.com/paper/2004.05937/) (2020)
- [Knowledge distillation: A good teacher is patient and consistent](https://mlsanity.com/paper/2106.05237/) (2021)
- [Knowledge Distillation from A Stronger Teacher](https://mlsanity.com/paper/2205.10536/) (2022)
- [Does Knowledge Distillation Really Work?](https://mlsanity.com/paper/2106.05945/) (2021)
- [On Mixup Regularization](https://mlsanity.com/paper/2006.06049/) (2020)
- [When and How Mixup Improves Calibration](https://mlsanity.com/paper/2102.06289/) (2021)
- [Mosaicking to Distill: Knowledge Distillation from Out-of-Domain Data](https://mlsanity.com/paper/2110.15094/) (2021)
- [A Survey of Mix-based Data Augmentation: Taxonomy, Methods, Applications, and Explainability](https://mlsanity.com/paper/2212.10888/) (2022)
- [MixSKD: Self-Knowledge Distillation from Mixup for Image Recognition](https://mlsanity.com/paper/2208.05768/) (2022)
- [Bayes Conditional Distribution Estimation for Knowledge Distillation Based on Conditional Mutual Information](https://mlsanity.com/paper/2401.08732/) (2024)
- [Computation-Efficient Knowledge Distillation via Uncertainty-Aware Mixup](https://mlsanity.com/paper/2012.09413/) (2020)
- [TAID: Temporally Adaptive Interpolated Distillation for Efficient Knowledge Transfer in Language Models](https://mlsanity.com/paper/2501.16937/) (2025)
- [A Survey on Mixup Augmentations and Beyond](https://mlsanity.com/paper/2409.05202/) (2024)
- [CKD: Contrastive Knowledge Distillation from A Sample-wise Perspective](https://mlsanity.com/paper/2404.14109/) (2024)
- [Understanding the Role of Mixup in Knowledge Distillation: An Empirical Study](https://mlsanity.com/paper/2211.03946/) (2022)
- [AuG-KD: Anchor-Based Mixup Generation for Out-of-Domain Knowledge Distillation](https://mlsanity.com/paper/2403.07030/) (2024)
- [SMILE: Self-Distilled MIxup for Efficient Transfer LEarning](https://mlsanity.com/paper/2103.13941/) (2021)
- [Dual-Forward Path Teacher Knowledge Distillation: Bridging the Capacity Gap Between Teacher and Student](https://mlsanity.com/paper/2506.18244/) (2025)

## Built on by (most cited first, 1 of 1)

- [Unknown-Traffic Detection, Calibration and Shortcut Reliance in Distilled Encrypted-Traffic Classifiers over One Year](https://mlsanity.com/paper/2609.31141/) (2026)

## Abstract (arXiv)

Knowledge Distillation (KD) and mixup have proven effective at inducing smoothness in class boundaries; KD captures inherent class relationships in probability distributions, and mixup enforces them through convex combinations of inputs. Their interaction, however, remains poorly understood, particularly when mixup is applied only during student training. In this setting, the teacher is queried on inputs drawn from a vicinal distribution it never saw during training, a controlled mismatch whose effect on knowledge transfer has not been characterised. We show that this mismatch causes the teacher's supervisory signal to be dominated by distributional confusion rather than inter-class structure. Despite it, the student does not merely imitate the teacher: it independently acquires greater linearity in the vicinal region, a structural property that the teacher lacks, and goes beyond dark-knowledge transfer. KD with mixup consistently improves student accuracy and reduces overconfidence by an order of magnitude relative to the baseline, across CIFAR and ImageNet with varying-capacity teachers. Crucially, calibration propagates from teacher to student independently of accuracy transfer, and temperature scaling governs a measurable accuracy-calibration trade-off that becomes more pronounced under vicinal training. These results reframe mixup distillation not as a degraded version of standard KD, but as a richer transfer channel that simultaneously shapes discriminative performance, uncertainty estimation, and representational geometry.