# Learning From Multiple Experts: Self-paced Knowledge Distillation for Long-tailed Classification

arXiv: https://arxiv.org/abs/2001.01536 · HTML: https://mlsanity.com/paper/2001.01536/

## Key facts

- Cited by 175 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 6 Jan 2020.

## Built on by (most cited first, 30 of 175)

- [Deep Long-Tailed Learning: A Survey](https://mlsanity.com/paper/2110.04596/) (2021)
- [Improving Calibration for Long-Tailed Recognition](https://mlsanity.com/paper/2104.00466/) (2021)
- [Contrast with Reconstruct: Contrastive 3D Representation Learning Guided by Generative Pretraining](https://mlsanity.com/paper/2302.02318/) (2023)
- [Curriculum Temperature for Knowledge Distillation](https://mlsanity.com/paper/2211.16231/) (2022)
- [Long-Tailed Recognition via Weight Balancing](https://mlsanity.com/paper/2203.14197/) (2022)
- [Retrieval Augmented Classification for Long-Tail Visual Recognition](https://mlsanity.com/paper/2202.11233/) (2022)
- [ResLT: Residual Learning for Long-tailed Recognition](https://mlsanity.com/paper/2101.10633/) (2021)
- [COTR: Compact Occupancy TRansformer for Vision-based 3D Occupancy Prediction](https://mlsanity.com/paper/2312.01919/) (2023)
- [Self-Supervised Aggregation of Diverse Experts for Test-Agnostic Long-Tailed Recognition](https://mlsanity.com/paper/2107.09249/) (2021)
- [ACE: Ally Complementary Experts for Solving Long-Tailed Recognition in One-Shot](https://mlsanity.com/paper/2108.02385/) (2021)
- [Distributional Robustness Loss for Long-tail Learning](https://mlsanity.com/paper/2104.03066/) (2021)
- [A Survey on Long-Tailed Visual Recognition](https://mlsanity.com/paper/2205.13775/) (2022)
- [Self Supervision to Distillation for Long-Tailed Visual Recognition](https://mlsanity.com/paper/2109.04075/) (2021)
- [Distilling Virtual Examples for Long-tailed Recognition](https://mlsanity.com/paper/2103.15042/) (2021)
- [Adaptive Methods for Real-World Domain Generalization](https://mlsanity.com/paper/2103.15796/) (2021)
- [Learning of Visual Relations: The Devil is in the Tails](https://mlsanity.com/paper/2108.09668/) (2021)
- [Nested Collaborative Learning for Long-Tailed Visual Recognition](https://mlsanity.com/paper/2203.15359/) (2022)
- [Federated Learning on Heterogeneous and Long-Tailed Data via Classifier Re-Training with Federated Features](https://mlsanity.com/paper/2204.13399/) (2022)
- [A Survey on Green Deep Learning](https://mlsanity.com/paper/2111.05193/) (2021)
- [DASO: Distribution-Aware Semantics-Oriented Pseudo-label for Imbalanced Semi-Supervised Learning](https://mlsanity.com/paper/2106.05682/) (2021)
- [Trustworthy Long-Tailed Classification](https://mlsanity.com/paper/2111.09030/) (2021)
- [Preserving Privacy in Federated Learning with Ensemble Cross-Domain Knowledge Distillation](https://mlsanity.com/paper/2209.04599/) (2022)
- [Partial and Asymmetric Contrastive Learning for Out-of-Distribution Detection in Long-Tailed Recognition](https://mlsanity.com/paper/2207.01160/) (2022)
- [Neural Collapse Inspired Attraction-Repulsion-Balanced Loss for Imbalanced Learning](https://mlsanity.com/paper/2204.08735/) (2022)
- [Probabilistic Contrastive Learning for Long-Tailed Visual Recognition](https://mlsanity.com/paper/2403.06726/) (2024)
- [TAM: Topology-Aware Margin Loss for Class-Imbalanced Node Classification](https://mlsanity.com/paper/2206.12917/) (2022)
- [Long-Tailed Class Incremental Learning](https://mlsanity.com/paper/2210.00266/) (2022)
- [Invariant Feature Learning for Generalized Long-Tailed Classification](https://mlsanity.com/paper/2207.09504/) (2022)
- [Learning Imbalanced Data with Vision Transformers](https://mlsanity.com/paper/2212.02015/) (2022)
- [Adaptive Logit Adjustment Loss for Long-Tailed Visual Recognition](https://mlsanity.com/paper/2104.06094/) (2021)

## Abstract (arXiv)

In real-world scenarios, data tends to exhibit a long-tailed distribution, which increases the difficulty of training deep networks. In this paper, we propose a novel self-paced knowledge distillation framework, termed Learning From Multiple Experts (LFME). Our method is inspired by the observation that networks trained on less imbalanced subsets of the distribution often yield better performances than their jointly-trained counterparts. We refer to these models as 'Experts', and the proposed LFME framework aggregates the knowledge from multiple 'Experts' to learn a unified student model. Specifically, the proposed framework involves two levels of adaptive learning schedules: Self-paced Expert Selection and Curriculum Instance Selection, so that the knowledge is adaptively transferred to the 'Student'. We conduct extensive experiments and demonstrate that our method is able to achieve superior performances compared to state-of-the-art methods. We also show that our method can be easily plugged into state-of-the-art long-tailed classification algorithms for further improvements.