# Self-Distillation from the Last Mini-Batch for Consistency Regularization

arXiv: https://arxiv.org/abs/2203.16172 · HTML: https://mlsanity.com/paper/2203.16172/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- Placed in General knowledge distillation methods, one of 12 directions in Knowledge distillation.
- Builds on 35 of its 44 references that are in the map.
- First posted on arXiv 30 Mar 2022.

## Placed in

- Knowledge distillation / [General knowledge distillation methods](https://mlsanity.com/distill/general-knowledge-distillation-methods/)

## Builds on (most cited first, 30 of 35)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [Wide Residual Networks](https://mlsanity.com/paper/1605.07146/) (2016)
- [CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Features](https://mlsanity.com/paper/1905.04899/) (2019)
- [FitNets: Hints for Thin Deep Nets](https://mlsanity.com/paper/1412.6550/) (2014)
- [Improved Regularization of Convolutional Neural Networks with Cutout](https://mlsanity.com/paper/1708.04552/) (2017)
- [Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfer](https://mlsanity.com/paper/1612.03928/) (2016)
- [Contrastive Representation Distillation](https://mlsanity.com/paper/1910.10699/) (2019)
- [Deep Mutual Learning](https://mlsanity.com/paper/1706.00384/) (2017)
- [Born Again Neural Networks](https://mlsanity.com/paper/1805.04770/) (2018)
- [Patient Knowledge Distillation for BERT Model Compression](https://mlsanity.com/paper/1908.09355/) (2019)
- [Be Your Own Teacher: Improve the Performance of Convolutional Neural Networks via Self Distillation](https://mlsanity.com/paper/1905.08094/) (2019)
- [Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results](https://mlsanity.com/paper/1703.01780/) (2017)
- [Unsupervised Label Noise Modeling and Loss Correction](https://mlsanity.com/paper/1904.11238/) (2019)
- [Training Deep Neural Networks on Noisy Labels with Bootstrapping](https://mlsanity.com/paper/1412.6596/) (2014)
- [Knowledge Distillation and Student-Teacher Learning for Visual Intelligence: A Review and New Outlooks](https://mlsanity.com/paper/2004.05937/) (2020)
- [Knowledge Distillation by On-the-Fly Native Ensemble](https://mlsanity.com/paper/1806.04606/) (2018)
- [Regularizing Class-wise Predictions via Self-knowledge Distillation](https://mlsanity.com/paper/2003.13964/) (2020)
- [Knowledge Distillation Meets Self-Supervision](https://mlsanity.com/paper/2006.07114/) (2020)
- [Online Knowledge Distillation with Diverse Peers](https://mlsanity.com/paper/1912.00350/) (2019)
- [Self-Knowledge Distillation with Progressive Refinement of Targets](https://mlsanity.com/paper/2006.12000/) (2020)
- [Snapshot Distillation: Teacher-Student Optimization in One Generation](https://mlsanity.com/paper/1812.00123/) (2018)
- [Delving Deep into Label Smoothing](https://mlsanity.com/paper/2011.12562/) (2020)
- [Peer Collaborative Learning for Online Knowledge Distillation](https://mlsanity.com/paper/2006.04147/) (2020)
- [Unsupervised Multi-Target Domain Adaptation Through Knowledge Distillation](https://mlsanity.com/paper/2007.07077/) (2020)
- [Analyzing the Noise Robustness of Deep Neural Networks](https://mlsanity.com/paper/2001.09395/) (2020)

## Built on by (most cited first, 29 of 29)

- [Categories of Response-Based, Feature-Based, and Relation-Based Knowledge Distillation](https://mlsanity.com/paper/2306.10687/) (2023)
- [A Comprehensive Review of Knowledge Distillation in Computer Vision](https://mlsanity.com/paper/2404.00936/) (2024)
- [CleaR: Towards Robust and Generalized Parameter-Efficient Fine-Tuning for Noisy Label Learning](https://mlsanity.com/paper/2411.00873/) (2024)
- [A Survey of Historical Learning: Learning Models with Learning History](https://mlsanity.com/paper/2303.12992/) (2023)
- [Boosting Residual Networks with Group Knowledge](https://mlsanity.com/paper/2308.13772/) (2023)
- [Double Reverse Regularization Network Based on Self-Knowledge Distillation for SAR Object Classification](https://mlsanity.com/paper/2311.15231/) (2023)
- [Self-Cooperation Knowledge Distillation for Novel Class Discovery](https://mlsanity.com/paper/2407.01930/) (2024)
- [Agentic-R1: Distilled Dual-Strategy Reasoning](https://mlsanity.com/paper/2507.05707/) (2025)
- [Enhanced Sparsification via Stimulative Training](https://mlsanity.com/paper/2403.06417/) (2024)
- [BASIC: Boosting Visual Alignment with Intrinsic Refined Embeddings in Multimodal Large Language Models](https://mlsanity.com/paper/2508.06895/) (2025)
- [AI-KD: Adversarial learning and Implicit regularization for self-Knowledge Distillation](https://mlsanity.com/paper/2211.10938/) (2022)
- [Lightweight Self-Knowledge Distillation with Multi-source Information Fusion](https://mlsanity.com/paper/2305.09183/) (2023)
- [Right Time to Learn:Promoting Generalization via Bio-inspired Spacing Effect in Knowledge Distillation](https://mlsanity.com/paper/2502.06192/) (2025)
- [Rethinking Random Masking in Self-Distillation on ViT](https://mlsanity.com/paper/2506.10582/) (2025)
- [Multi-Teacher Knowledge Distillation via Teacher-Informed Mixture Priors](https://mlsanity.com/paper/2605.27967/) (2026)
- [Self-discipline on multiple channels](https://mlsanity.com/paper/2304.14224/) (2023)
- [Dynamic Prototype Adaptation with Distillation for Few-shot Point Cloud Segmentation](https://mlsanity.com/paper/2401.16051/) (2024)
- [S2HPruner: Soft-to-Hard Distillation Bridges the Discretization Gap in Pruning](https://mlsanity.com/paper/2410.07046/) (2024)
- [Reinforced Imitative Trajectory Planning for Urban Automated Driving](https://mlsanity.com/paper/2410.15607/) (2024)
- [Dynamic Self-Distillation via Previous Mini-batches for Fine-tuning Small Language Models](https://mlsanity.com/paper/2411.16991/) (2024)
- [sDREAMER: Self-distilled Mixture-of-Modality-Experts Transformer for Automatic Sleep Staging](https://mlsanity.com/paper/2501.16329/) (2025)
- [Intra-class Patch Swap for Self-Distillation](https://mlsanity.com/paper/2505.14124/) (2025)
- [The Power of Certainty: How Confident Models Lead to Better Segmentation](https://mlsanity.com/paper/2507.10490/) (2025)
- [Joint graph entropy knowledge distillation for point cloud classification and robustness against corruptions](https://mlsanity.com/paper/2509.22150/) (2025)
- [Desensitizing for Improving Corruption Robustness in Point Cloud Classification through Adversarial Training](https://mlsanity.com/paper/2509.23010/) (2025)
- [Deep Probabilistic Supervision for Image Classification](https://mlsanity.com/paper/2512.24162/) (2025)
- [Teacher-Guided Student Self-Knowledge Distillation Using Diffusion Model](https://mlsanity.com/paper/2602.02107/) (2026)
- [VISTA: Validation-Informed Trajectory Adaptation via Self-Distillation](https://mlsanity.com/paper/2604.12044/) (2026)
- [TVTA: Trajectory-Aware Viseme-Guided Temporal Aggregation for Event-Based Lip Reading](https://mlsanity.com/paper/2607.08236/) (2026)

## Abstract (arXiv)

Knowledge distillation (KD) shows a bright promise as a powerful regularization strategy to boost generalization ability by leveraging learned sample-level soft targets. Yet, employing a complex pre-trained teacher network or an ensemble of peer students in existing KD is both time-consuming and computationally costly. Various self KD methods have been proposed to achieve higher distillation efficiency. However, they either require extra network architecture modification or are difficult to parallelize. To cope with these challenges, we propose an efficient and reliable self-distillation framework, named Self-Distillation from Last Mini-Batch (DLB). Specifically, we rearrange the sequential sampling by constraining half of each mini-batch coinciding with the previous iteration. Meanwhile, the rest half will coincide with the upcoming iteration. Afterwards, the former half mini-batch distills on-the-fly soft targets generated in the previous iteration. Our proposed mechanism guides the training stability and consistency, resulting in robustness to label noise. Moreover, our method is easy to implement, without taking up extra run-time memory or requiring model structure modification. Experimental results on three classification benchmarks illustrate that our approach can consistently outperform state-of-the-art self-distillation approaches with different network architectures. Additionally, our method shows strong compatibility with augmentation strategies by gaining additional performance improvement. The code is available at https://github.com/Meta-knowledge-Lab/DLB.