# MEAL V2: Boosting Vanilla ResNet-50 to 80%+ Top-1 Accuracy on ImageNet without Tricks

arXiv: https://arxiv.org/abs/2009.08453 · HTML: https://mlsanity.com/paper/2009.08453/

## Key facts

- Cited by 40 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 17 Sep 2020.

## Built on by (most cited first, 30 of 40)

- [Knowledge distillation: A good teacher is patient and consistent](https://mlsanity.com/paper/2106.05237/) (2021)
- [SEED: Self-supervised Distillation For Visual Representation](https://mlsanity.com/paper/2101.04731/) (2021)
- [MobileOne: An Improved One millisecond Mobile Backbone](https://mlsanity.com/paper/2206.04040/) (2022)
- [Masked Video Distillation: Rethinking Masked Feature Modeling for Self-supervised Video Representation Learning](https://mlsanity.com/paper/2212.04500/) (2022)
- [Dynamic Slimmable Network](https://mlsanity.com/paper/2103.13258/) (2021)
- [Vision Transformer Slimming: Multi-Dimension Searching in Continuous Optimization Space](https://mlsanity.com/paper/2201.00814/) (2022)
- [BossNAS: Exploring Hybrid CNN-transformers with Block-wisely Self-supervised Neural Architecture Search](https://mlsanity.com/paper/2103.12424/) (2021)
- [Is Label Smoothing Truly Incompatible with Knowledge Distillation: An Empirical Study](https://mlsanity.com/paper/2104.00676/) (2021)
- [A Fast Knowledge Distillation Framework for Visual Recognition](https://mlsanity.com/paper/2112.01528/) (2021)
- [Towards Understanding and Boosting Adversarial Transferability from a Distribution Perspective](https://mlsanity.com/paper/2210.04213/) (2022)
- [Student Network Learning via Evolutionary Knowledge Distillation](https://mlsanity.com/paper/2103.13811/) (2021)
- [Multimodal Distillation for Egocentric Action Recognition](https://mlsanity.com/paper/2307.07483/) (2023)
- [Sliced Recursive Transformer](https://mlsanity.com/paper/2111.05297/) (2021)
- [Densely Nested Top-Down Flows for Salient Object Detection](https://mlsanity.com/paper/2102.09133/) (2021)
- [CLIP-CID: Efficient CLIP Distillation via Cluster-Instance Discrimination](https://mlsanity.com/paper/2408.09441/) (2024)
- [Active Data Curation Effectively Distills Large-Scale Multimodal Models](https://mlsanity.com/paper/2411.18674/) (2024)
- [ERM++: An Improved Baseline for Domain Generalization](https://mlsanity.com/paper/2304.01973/) (2023)
- [S2-BNN: Bridging the Gap Between Self-Supervised Real and 1-bit Neural Networks via Guided Distribution Calibration](https://mlsanity.com/paper/2102.08946/) (2021)
- [Join the High Accuracy Club on ImageNet with A Binary Neural Network Ticket](https://mlsanity.com/paper/2211.12933/) (2022)
- [Cosine Model Watermarking Against Ensemble Distillation](https://mlsanity.com/paper/2203.02777/) (2022)
- [Knowledge Distillation of Transformer-based Language Models Revisited](https://mlsanity.com/paper/2206.14366/) (2022)
- [Reinforce Data, Multiply Impact: Improved Model Accuracy and Robustness with Dataset Reinforcement](https://mlsanity.com/paper/2303.08983/) (2023)
- [What Makes a Good Dataset for Knowledge Distillation?](https://mlsanity.com/paper/2411.12817/) (2024)
- [Progressive Learning without Forgetting](https://mlsanity.com/paper/2211.15215/) (2022)
- [Rethinking Out-of-Distribution Detection From a Human-Centric Perspective](https://mlsanity.com/paper/2211.16778/) (2022)
- [DS-Net++: Dynamic Weight Slicing for Efficient Inference in CNNs and Transformers](https://mlsanity.com/paper/2109.10060/) (2021)
- [FerKD: Surgical Label Adaptation for Efficient Distillation](https://mlsanity.com/paper/2312.17473/) (2023)
- [Cross-modal Contrastive Distillation for Instructional Activity Anticipation](https://mlsanity.com/paper/2201.06734/) (2022)
- [Efficient Knowledge Distillation: Empowering Small Language Models with Teacher Model Insights](https://mlsanity.com/paper/2409.12586/) (2024)
- [TransBoost: Improving the Best ImageNet Performance using Deep Transduction](https://mlsanity.com/paper/2205.13331/) (2022)

## Abstract (arXiv)

We introduce a simple yet effective distillation framework that is able to boost the vanilla ResNet-50 to 80%+ Top-1 accuracy on ImageNet without tricks. We construct such a framework through analyzing the problems in the existing classification system and simplify the base method ensemble knowledge distillation via discriminators by: (1) adopting the similarity loss and discriminator only on the final outputs and (2) using the average of softmax probabilities from all teacher ensembles as the stronger supervision. Intriguingly, three novel perspectives are presented for distillation: (1) weight decay can be weakened or even completely removed since the soft label also has a regularization effect; (2) using a good initialization for students is critical; and (3) one-hot/hard label is not necessary in the distillation process if the weights are well initialized. We show that such a straight-forward framework can achieve state-of-the-art results without involving any commonly-used techniques, such as architecture modification; outside training data beyond ImageNet; autoaug/randaug; cosine learning rate; mixup/cutmix training; label smoothing; etc. Our method obtains 80.67% top-1 accuracy on ImageNet using a single crop-size of 224x224 with vanilla ResNet-50, outperforming the previous state-of-the-arts by a significant margin under the same network structure. Our result can be regarded as a strong baseline using knowledge distillation, and to our best knowledge, this is also the first method that is able to boost vanilla ResNet-50 to surpass 80% on ImageNet without architecture modification or additional training data. On smaller ResNet-18, our distillation framework consistently improves from 69.76% to 73.19%, which shows tremendous practical values in real-world applications. Our code and models are available at: https://github.com/szq0214/MEAL-V2.