# Label Refinery: Improving ImageNet Classification through Label Progression

arXiv: https://arxiv.org/abs/1805.02641 · HTML: https://mlsanity.com/paper/1805.02641/

## Key facts

- Cited by 65 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 May 2018.

## Built on by (most cited first, 30 of 65)

- [Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time](https://mlsanity.com/paper/2203.05482/) (2022)
- [Neural Attention Distillation: Erasing Backdoor Triggers from Deep Neural Networks](https://mlsanity.com/paper/2101.05930/) (2021)
- [Novel Visual Category Discovery with Dual Ranking Statistics and Mutual Knowledge Distillation](https://mlsanity.com/paper/2107.03358/) (2021)
- [Is Label Smoothing Truly Incompatible with Knowledge Distillation: An Empirical Study](https://mlsanity.com/paper/2104.00676/) (2021)
- [Is one annotation enough? A data-centric image classification benchmark for noisy and ambiguous label estimation](https://mlsanity.com/paper/2207.06214/) (2022)
- [Data Efficient Language-supervised Zero-shot Recognition with Optimal Transport Distillation](https://mlsanity.com/paper/2112.09445/) (2021)
- [How many Observations are Enough? Knowledge Distillation for Trajectory Forecasting](https://mlsanity.com/paper/2203.04781/) (2022)
- [Sliced Recursive Transformer](https://mlsanity.com/paper/2111.05297/) (2021)
- [Teacher-Student Architecture for Knowledge Distillation: A Survey](https://mlsanity.com/paper/2308.04268/) (2023)
- [Teacher-Student Architecture for Knowledge Learning: A Survey](https://mlsanity.com/paper/2210.17332/) (2022)
- [Data-Efficient Language-Supervised Zero-Shot Learning with Self-Distillation](https://mlsanity.com/paper/2104.08945/) (2021)
- [Object Localization under Single Coarse Point Supervision](https://mlsanity.com/paper/2203.09338/) (2022)
- [On the benefits of knowledge distillation for adversarial robustness](https://mlsanity.com/paper/2203.07159/) (2022)
- [Self-Adaptive Training: Bridging Supervised and Self-Supervised Learning](https://mlsanity.com/paper/2101.08732/) (2021)
- [Text is Text, No Matter What: Unifying Text Recognition using Knowledge Distillation](https://mlsanity.com/paper/2107.12087/) (2021)
- [Energy-constrained Self-training for Unsupervised Domain Adaptation](https://mlsanity.com/paper/2101.00316/) (2021)
- [Self-distillation with Batch Knowledge Ensembling Improves ImageNet Classification](https://mlsanity.com/paper/2104.13298/) (2021)
- [Few-Shot Learning with a Strong Teacher](https://mlsanity.com/paper/2107.00197/) (2021)
- [Exploiting Unlabelled Photos for Stronger Fine-Grained SBIR](https://mlsanity.com/paper/2303.13779/) (2023)
- [Generalized Knowledge Distillation via Relationship Matching](https://mlsanity.com/paper/2205.01915/) (2022)
- [A Survey of Automated Data Augmentation Algorithms for Deep Learning-based Image Classification Tasks](https://mlsanity.com/paper/2206.06544/) (2022)
- [Complementary Bi-directional Feature Compression for Indoor 360deg Semantic Segmentation with Self-distillation](https://mlsanity.com/paper/2207.02437/) (2022)
- [Efficient One Pass Self-distillation with Zipf's Label Smoothing](https://mlsanity.com/paper/2207.12980/) (2022)
- [KV Prediction for Improved Time to First Token](https://mlsanity.com/paper/2410.08391/) (2024)
- [Physical Knowledge Enhanced Deep Neural Network for Sea Surface Temperature Prediction](https://mlsanity.com/paper/2304.09376/) (2023)
- [ISP Distillation](https://mlsanity.com/paper/2101.10203/) (2021)
- [A Theory-Driven Self-Labeling Refinement Method for Contrastive Representation Learning](https://mlsanity.com/paper/2106.14749/) (2021)
- [Unsupervised Domain Adaptation for RF-based Gesture Recognition](https://mlsanity.com/paper/2111.10602/) (2021)
- [Constrained Mean Shift Using Distant Yet Related Neighbors for Representation Learning](https://mlsanity.com/paper/2112.04607/) (2021)
- [Boosting the Adversarial Transferability of Surrogate Models with Dark Knowledge](https://mlsanity.com/paper/2206.08316/) (2022)

## Abstract (arXiv)

Among the three main components (data, labels, and models) of any supervised learning system, data and models have been the main subjects of active research. However, studying labels and their properties has received very little attention. Current principles and paradigms of labeling impose several challenges to machine learning algorithms. Labels are often incomplete, ambiguous, and redundant. In this paper we study the effects of various properties of labels and introduce the Label Refinery: an iterative procedure that updates the ground truth labels after examining the entire dataset. We show significant gain using refined labels across a wide range of models. Using a Label Refinery improves the state-of-the-art top-1 accuracy of (1) AlexNet from 59.3 to 67.2, (2) MobileNet from 70.6 to 73.39, (3) MobileNet-0.25 from 50.6 to 55.59, (4) VGG19 from 72.7 to 75.46, and (5) Darknet19 from 72.9 to 74.47.