# Class-Balanced Distillation for Long-Tailed Visual Recognition

arXiv: https://arxiv.org/abs/2104.05279 · HTML: https://mlsanity.com/paper/2104.05279/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- Builds on 51 of its 67 references that are in the map.
- First posted on arXiv 12 Apr 2021.

## Builds on (most cited first, 30 of 51)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1802.02611/) (2018)
- [Prototypical Networks for Few-shot Learning](https://mlsanity.com/paper/1703.05175/) (2017)
- [Class-Balanced Loss Based on Effective Number of Samples](https://mlsanity.com/paper/1901.05555/) (2019)
- [LVIS: A Dataset for Large Vocabulary Instance Segmentation](https://mlsanity.com/paper/1908.03195/) (2019)
- [Temporal Ensembling for Semi-Supervised Learning](https://mlsanity.com/paper/1610.02242/) (2016)
- [Deep Clustering for Unsupervised Learning of Visual Features](https://mlsanity.com/paper/1807.05520/) (2018)
- [Learning Imbalanced Datasets with Label-Distribution-Aware Margin Loss](https://mlsanity.com/paper/1906.07413/) (2019)
- [Decoupling Representation and Classifier for Long-Tailed Recognition](https://mlsanity.com/paper/1910.09217/) (2019)
- [Large-Scale Long-Tailed Recognition in an Open World](https://mlsanity.com/paper/1904.05160/) (2019)
- [Long-tail learning via logit adjustment](https://mlsanity.com/paper/2007.07314/) (2020)
- [BBN: Bilateral-Branch Network with Cumulative Learning for Long-Tailed Visual Recognition](https://mlsanity.com/paper/1912.02413/) (2019)
- [Balanced Meta-Softmax for Long-Tailed Visual Recognition](https://mlsanity.com/paper/2007.10740/) (2020)
- [Dynamic Few-Shot Visual Learning without Forgetting](https://mlsanity.com/paper/1804.09458/) (2018)
- [Google Landmarks Dataset v2 -- A Large-Scale Benchmark for Instance-Level Recognition and Retrieval](https://mlsanity.com/paper/2004.01804/) (2020)
- [Label Propagation for Deep Semi-supervised Learning](https://mlsanity.com/paper/1904.04717/) (2019)
- [Equalization Loss for Long-Tailed Object Recognition](https://mlsanity.com/paper/2003.05176/) (2020)
- [Long-Tailed Classification by Keeping the Good and Removing the Bad Momentum Causal Effect](https://mlsanity.com/paper/2009.12991/) (2020)
- [Large-Scale Image Retrieval with Attentive Deep Local Features](https://mlsanity.com/paper/1612.06321/) (2016)
- [Rethinking the Value of Labels for Improving Class-Imbalanced Learning](https://mlsanity.com/paper/2006.07529/) (2020)
- [Revisiting Oxford and Paris: Large-Scale Image Retrieval Benchmarking](https://mlsanity.com/paper/1803.11285/) (2018)
- [Learning From Multiple Experts: Self-paced Knowledge Distillation for Long-tailed Classification](https://mlsanity.com/paper/2001.01536/) (2020)
- [Large Scale Fine-Grained Categorization and Domain-Specific Transfer Learning](https://mlsanity.com/paper/1806.06193/) (2018)

## Built on by (most cited first, 27 of 27)

- [LPT: Long-tailed Prompt Tuning for Image Classification](https://mlsanity.com/paper/2210.01033/) (2022)
- [Learning Imbalanced Data with Vision Transformers](https://mlsanity.com/paper/2212.02015/) (2022)
- [A Systematic Review on Long-Tailed Learning](https://mlsanity.com/paper/2408.00483/) (2024)
- [Subclass-balancing Contrastive Learning for Long-tailed Recognition](https://mlsanity.com/paper/2306.15925/) (2023)
- [Teacher's pet: understanding and mitigating biases in distillation](https://mlsanity.com/paper/2106.10494/) (2021)
- [Cross-Domain Empirical Risk Minimization for Unbiased Long-tailed Classification](https://mlsanity.com/paper/2112.14380/) (2021)
- [Long-tail Recognition via Compositional Knowledge Transfer](https://mlsanity.com/paper/2112.06741/) (2021)
- [DeiT-LT Distillation Strikes Back for Vision Transformer Training on Long-Tailed Datasets](https://mlsanity.com/paper/2404.02900/) (2024)
- [Decoupled Training for Long-Tailed Classification With Stochastic Representations](https://mlsanity.com/paper/2304.09426/) (2023)
- [Hierarchical Knowledge Guided Learning for Real-world Retinal Diseases Recognition](https://mlsanity.com/paper/2111.08913/) (2021)
- [Tackling Long-Tailed Category Distribution Under Domain Shifts](https://mlsanity.com/paper/2207.10150/) (2022)
- [Feature Generation for Long-tail Classification](https://mlsanity.com/paper/2111.05956/) (2021)
- [Transfer and Share: Semi-Supervised Learning from Long-Tailed Data](https://mlsanity.com/paper/2205.13358/) (2022)
- [Inverse Image Frequency for Long-tailed Image Recognition](https://mlsanity.com/paper/2209.04861/) (2022)
- [Adaptive Parametric Activation: Unifying and Generalising Activation Functions Across Tasks](https://mlsanity.com/paper/2407.08567/) (2024)
- [Improving Fairness in Large-Scale Object Recognition by CrowdSourced Demographic Information](https://mlsanity.com/paper/2206.01326/) (2022)
- [Set Learning for Accurate and Calibrated Models](https://mlsanity.com/paper/2307.02245/) (2023)
- [Learn from Balance: Rectifying Knowledge Transfer for Long-Tailed Scenarios](https://mlsanity.com/paper/2409.07694/) (2024)
- [Enhancing Features in Long-tailed Data Using Large Vision Model](https://mlsanity.com/paper/2504.10852/) (2025)
- [Towards Effective Collaborative Learning in Long-Tailed Recognition](https://mlsanity.com/paper/2305.03378/) (2023)
- [Gradient-Aware Logit Adjustment Loss for Long-tailed Classifier](https://mlsanity.com/paper/2403.09036/) (2024)
- [Dynamic Feature Learning and Matching for Class-Incremental Learning](https://mlsanity.com/paper/2405.08533/) (2024)
- [Rethinking Meta-Learning from a Learning Lens](https://mlsanity.com/paper/2409.08474/) (2024)
- [Propheter: Prophetic Teacher Guided Long-Tailed Distribution Learning](https://mlsanity.com/paper/2304.04135/) (2023)
- [Robust Domain Generalization under Divergent Marginal and Conditional Distributions](https://mlsanity.com/paper/2602.02015/) (2026)
- [Balancing Knowledge Distillation for Imbalance Learning with Bilevel Optimization](https://mlsanity.com/paper/2605.17839/) (2026)
- [DAMEL: Dual-Axis Multi-Expert Learning for Class-Imbalanced Learning](https://mlsanity.com/paper/2605.30135/) (2026)

## Abstract (arXiv)

Real-world imagery is often characterized by a significant imbalance of the number of images per class, leading to long-tailed distributions. An effective and simple approach to long-tailed visual recognition is to learn feature representations and a classifier separately, with instance and class-balanced sampling, respectively. In this work, we introduce a new framework, by making the key observation that a feature representation learned with instance sampling is far from optimal in a long-tailed setting. Our main contribution is a new training method, referred to as Class-Balanced Distillation (CBD), that leverages knowledge distillation to enhance feature representations. CBD allows the feature representation to evolve in the second training stage, guided by the teacher learned in the first stage. The second stage uses class-balanced sampling, in order to focus on under-represented classes. This framework can naturally accommodate the usage of multiple teachers, unlocking the information from an ensemble of models to enhance recognition capabilities. Our experiments show that the proposed technique consistently outperforms the state of the art on long-tailed recognition benchmarks such as ImageNet-LT, iNaturalist17 and iNaturalist18.