# Distilling Virtual Examples for Long-tailed Recognition

arXiv: https://arxiv.org/abs/2103.15042 · HTML: https://mlsanity.com/paper/2103.15042/

## Key facts

- Cited by 72 papers in the map (data through 14 Oct 2026).
- Placed in General knowledge distillation methods, one of 12 directions in Knowledge distillation.
- Builds on 26 of its 37 references that are in the map.
- First posted on arXiv 28 Mar 2021.

## Placed in

- Knowledge distillation / [General knowledge distillation methods](https://mlsanity.com/distill/general-knowledge-distillation-methods/)

## Builds on (most cited first, 26 of 26)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [Aggregated Residual Transformations for Deep Neural Networks](https://mlsanity.com/paper/1611.05431/) (2016)
- [Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour](https://mlsanity.com/paper/1706.02677/) (2017)
- [Class-Balanced Loss Based on Effective Number of Samples](https://mlsanity.com/paper/1901.05555/) (2019)
- [LVIS: A Dataset for Large Vocabulary Instance Segmentation](https://mlsanity.com/paper/1908.03195/) (2019)
- [Learning Imbalanced Datasets with Label-Distribution-Aware Margin Loss](https://mlsanity.com/paper/1906.07413/) (2019)
- [Decoupling Representation and Classifier for Long-Tailed Recognition](https://mlsanity.com/paper/1910.09217/) (2019)
- [Large-Scale Long-Tailed Recognition in an Open World](https://mlsanity.com/paper/1904.05160/) (2019)
- [Long-tail learning via logit adjustment](https://mlsanity.com/paper/2007.07314/) (2020)
- [BBN: Bilateral-Branch Network with Cumulative Learning for Long-Tailed Visual Recognition](https://mlsanity.com/paper/1912.02413/) (2019)
- [Balanced Meta-Softmax for Long-Tailed Visual Recognition](https://mlsanity.com/paper/2007.10740/) (2020)
- [Equalization Loss for Long-Tailed Object Recognition](https://mlsanity.com/paper/2003.05176/) (2020)
- [Long-tailed Recognition by Routing Diverse Distribution-Aware Experts](https://mlsanity.com/paper/2010.01809/) (2020)
- [Long-Tailed Classification by Keeping the Good and Removing the Bad Momentum Causal Effect](https://mlsanity.com/paper/2009.12991/) (2020)
- [Rethinking the Value of Labels for Improving Class-Imbalanced Learning](https://mlsanity.com/paper/2006.07529/) (2020)
- [Learning From Multiple Experts: Self-paced Knowledge Distillation for Long-tailed Classification](https://mlsanity.com/paper/2001.01536/) (2020)
- [Large Scale Fine-Grained Categorization and Domain-Specific Transfer Learning](https://mlsanity.com/paper/1806.06193/) (2018)
- [Disentangling Label Distribution for Long-tailed Visual Recognition](https://mlsanity.com/paper/2012.00321/) (2020)
- [Rethinking Class-Balanced Methods for Long-Tailed Visual Recognition from a Domain Adaptation Perspective](https://mlsanity.com/paper/2003.10780/) (2020)
- [Label Distribution Learning](https://mlsanity.com/paper/1408.6027/) (2014)
- [Relay Backpropagation for Effective Learning of Deep Convolutional Neural Networks](https://mlsanity.com/paper/1512.05830/) (2015)
- [Deep Label Distribution Learning with Label Ambiguity](https://mlsanity.com/paper/1611.01731/) (2016)
- [Background Splitting: Finding Rare Classes in a Sea of Background](https://mlsanity.com/paper/2008.12873/) (2020)

## Built on by (most cited first, 30 of 72)

- [Deep Long-Tailed Learning: A Survey](https://mlsanity.com/paper/2110.04596/) (2021)
- [Long-Tailed Recognition via Weight Balancing](https://mlsanity.com/paper/2203.14197/) (2022)
- [A Survey on Long-Tailed Visual Recognition](https://mlsanity.com/paper/2205.13775/) (2022)
- [DELTA: degradation-free fully test-time adaptation](https://mlsanity.com/paper/2301.13018/) (2023)
- [Nested Collaborative Learning for Long-Tailed Visual Recognition](https://mlsanity.com/paper/2203.15359/) (2022)
- [Long-Tail Learning with Foundation Model: Heavy Fine-Tuning Hurts](https://mlsanity.com/paper/2309.10019/) (2023)
- [Probabilistic Contrastive Learning for Long-Tailed Visual Recognition](https://mlsanity.com/paper/2403.06726/) (2024)
- [Invariant Feature Learning for Generalized Long-Tailed Classification](https://mlsanity.com/paper/2207.09504/) (2022)
- [Learning Imbalanced Data with Vision Transformers](https://mlsanity.com/paper/2212.02015/) (2022)
- [ImbSAM: A Closer Look at Sharpness-Aware Minimization in Class-Imbalanced Recognition](https://mlsanity.com/paper/2308.07815/) (2023)
- [BatchFormer: Learning to Explore Sample Relationships for Robust Representation Learning](https://mlsanity.com/paper/2203.01522/) (2022)
- [Escaping Saddle Points for Effective Generalization on Class-Imbalanced Data](https://mlsanity.com/paper/2212.13827/) (2022)
- [A Systematic Review on Long-Tailed Learning](https://mlsanity.com/paper/2408.00483/) (2024)
- [Introspective Distillation for Robust Question Answering](https://mlsanity.com/paper/2111.01026/) (2021)
- [Constructing Balance from Imbalance for Long-tailed Image Recognition](https://mlsanity.com/paper/2208.02567/) (2022)
- [A Unified Perspective for Loss-Oriented Imbalanced Learning via Localization](https://mlsanity.com/paper/2310.04752/) (2023)
- [Delving into Semantic Scale Imbalance](https://mlsanity.com/paper/2212.14613/) (2022)
- [When Noisy Labels Meet Long Tail Dilemmas: A Representation Calibration Method](https://mlsanity.com/paper/2211.10955/) (2022)
- [Balanced Product of Calibrated Experts for Long-Tailed Recognition](https://mlsanity.com/paper/2206.05260/) (2022)
- [Cross-Domain Empirical Risk Minimization for Unbiased Long-tailed Classification](https://mlsanity.com/paper/2112.14380/) (2021)
- [NICEST: Noisy Label Correction and Training for Robust Scene Graph Generation](https://mlsanity.com/paper/2207.13316/) (2022)
- [Relieving Long-tailed Instance Segmentation via Pairwise Class Balance](https://mlsanity.com/paper/2201.02784/) (2022)
- [Label-Noise Learning with Intrinsically Long-Tailed Data](https://mlsanity.com/paper/2208.09833/) (2022)
- [Billion-user Customer Lifetime Value Prediction: An Industrial-scale Solution from Kuaishou](https://mlsanity.com/paper/2208.13358/) (2022)
- [Quantized Feature Distillation for Network Quantization](https://mlsanity.com/paper/2307.10638/) (2023)
- [No One Left Behind: Improving the Worst Categories in Long-Tailed Learning](https://mlsanity.com/paper/2303.03630/) (2023)
- [NCL++: Nested Collaborative Learning for Long-Tailed Visual Recognition](https://mlsanity.com/paper/2306.16709/) (2023)
- [Focal-SAM: Focal Sharpness-Aware Minimization for Long-Tailed Classification](https://mlsanity.com/paper/2505.01660/) (2025)
- [Transfer and Share: Semi-Supervised Learning from Long-Tailed Data](https://mlsanity.com/paper/2205.13358/) (2022)
- [Predicting and Enhancing the Fairness of DNNs with the Curvature of Perceptual Manifolds](https://mlsanity.com/paper/2303.12307/) (2023)

## Abstract (arXiv)

We tackle the long-tailed visual recognition problem from the knowledge distillation perspective by proposing a Distill the Virtual Examples (DiVE) method. Specifically, by treating the predictions of a teacher model as virtual examples, we prove that distilling from these virtual examples is equivalent to label distribution learning under certain constraints. We show that when the virtual example distribution becomes flatter than the original input distribution, the under-represented tail classes will receive significant improvements, which is crucial in long-tailed recognition. The proposed DiVE method can explicitly tune the virtual example distribution to become flat. Extensive experiments on three benchmark datasets, including the large-scale iNaturalist ones, justify that the proposed DiVE method can significantly outperform state-of-the-art methods. Furthermore, additional analyses and experiments verify the virtual example interpretation, and demonstrate the effectiveness of tailored designs in DiVE for long-tailed problems.