# Relational Subsets Knowledge Distillation for Long-tailed Retinal Diseases Recognition

arXiv: https://arxiv.org/abs/2104.11057 · HTML: https://mlsanity.com/paper/2104.11057/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- Placed in Distillation for medical imaging, one of 12 directions in Knowledge distillation.
- Builds on 9 of its 14 references that are in the map.
- First posted on arXiv 22 Apr 2021.

## Placed in

- Knowledge distillation / [Distillation for medical imaging](https://mlsanity.com/distill/distillation-for-medical-imaging/)

## Builds on (most cited first, 9 of 9)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Learning Imbalanced Datasets with Label-Distribution-Aware Margin Loss](https://mlsanity.com/paper/1906.07413/) (2019)
- [Large-Scale Long-Tailed Recognition in an Open World](https://mlsanity.com/paper/1904.05160/) (2019)
- [Learning From Multiple Experts: Self-paced Knowledge Distillation for Long-tailed Classification](https://mlsanity.com/paper/2001.01536/) (2020)
- [Distribution-Balanced Loss for Multi-Label Classification in Long-Tailed Datasets](https://mlsanity.com/paper/2007.09654/) (2020)
- [Disc-aware Ensemble Network for Glaucoma Screening from Fundus Image](https://mlsanity.com/paper/1805.07549/) (2018)
- [Automatic detection of rare pathologies in fundus photographs using few-shot learning](https://mlsanity.com/paper/1907.09449/) (2019)
- [Improving Medical Image Classification with Label Noise Using Dual-uncertainty Estimation](https://mlsanity.com/paper/2103.00528/) (2021)
- [Subset Feature Learning for Fine-Grained Category Classification](https://mlsanity.com/paper/1505.02269/) (2015)

## Built on by (most cited first, 27 of 27)

- [Deep Long-Tailed Learning: A Survey](https://mlsanity.com/paper/2110.04596/) (2021)
- [Super-CLEVR: A Virtual Benchmark to Diagnose Domain Robustness in Visual Reasoning](https://mlsanity.com/paper/2212.00259/) (2022)
- [Long-Tailed Classification of Thorax Diseases on Chest X-Ray: A New Benchmark Study](https://mlsanity.com/paper/2208.13365/) (2022)
- [Towards long-tailed, multi-label disease classification from chest X-ray: Overview of the CXR-LT challenge](https://mlsanity.com/paper/2310.16112/) (2023)
- [On the Effectiveness of Out-of-Distribution Data in Self-Supervised Long-Tail Learning](https://mlsanity.com/paper/2306.04934/) (2023)
- [Hierarchical Knowledge Guided Learning for Real-world Retinal Diseases Recognition](https://mlsanity.com/paper/2111.08913/) (2021)
- [Distilling Long-tailed Datasets](https://mlsanity.com/paper/2408.14506/) (2024)
- [CT Image Synthesis Using Weakly Supervised Segmentation and Geometric Inter-Label Relations For COVID Image Analysis](https://mlsanity.com/paper/2106.10230/) (2021)
- [Flexible Sampling for Long-tailed Skin Lesion Classification](https://mlsanity.com/paper/2204.03161/) (2022)
- [DELTA: Decoupling Long-Tailed Online Continual Learning](https://mlsanity.com/paper/2404.04476/) (2024)
- [Towards Heterogeneous Long-tailed Learning: Benchmarking, Metrics, and Toolbox](https://mlsanity.com/paper/2307.08235/) (2023)
- [Learning of Inter-Label Geometric Relationships Using Self-Supervised Learning: Application To Gleason Grade Segmentation](https://mlsanity.com/paper/2110.00404/) (2021)
- [Unsupervised Domain Adaptation Using Feature Disentanglement And GCNs For Medical Image Classification](https://mlsanity.com/paper/2206.13123/) (2022)
- [A Survey of Dataset Refinement for Problems in Computer Vision Datasets](https://mlsanity.com/paper/2210.11717/) (2022)
- [Multimodal Generalized Zero Shot Learning for Gleason Grading using Self-Supervised Learning](https://mlsanity.com/paper/2111.07646/) (2021)
- [Interpretable Few-Shot Retinal Disease Diagnosis with Concept-Guided Prompting of Vision-Language Models](https://mlsanity.com/paper/2503.02917/) (2025)
- [Group Distributionally Robust Knowledge Distillation](https://mlsanity.com/paper/2311.00476/) (2023)
- [MONICA: Benchmarking on Long-tailed Medical Image Classification](https://mlsanity.com/paper/2410.02010/) (2024)
- [Overview of the CXR-LT 2026 Challenge: Multi-Center Long-Tailed and Zero Shot Chest X-ray Classification](https://mlsanity.com/paper/2602.22092/) (2026)
- [Interpretable Saliency Maps And Self-Supervised Learning For Generalized Zero Shot Medical Image Classification](https://mlsanity.com/paper/2204.01728/) (2022)
- [Enhancing Features in Long-tailed Data Using Large Vision Model](https://mlsanity.com/paper/2504.10852/) (2025)
- [Improved Super Resolution of MR Images Using CNNs and Vision Transformers](https://mlsanity.com/paper/2207.11748/) (2022)
- [Generalized Cross-domain Multi-label Few-shot Learning for Chest X-rays](https://mlsanity.com/paper/2309.04462/) (2023)
- [GazeLT: Visual attention-guided long-tailed disease classification in chest radiographs](https://mlsanity.com/paper/2508.09478/) (2025)
- [CXR-LT 2026 Challenge: Multi-Center Long-Tailed and Zero Shot Chest X-ray Classification](https://mlsanity.com/paper/2604.15555/) (2026)
- [Cyclic Generative Adversarial Networks With Congruent Image-Report Generation For Explainable Medical Image Analysis](https://mlsanity.com/paper/2211.08424/) (2022)
- [Distilling Balanced Knowledge from a Biased Teacher](https://mlsanity.com/paper/2506.18496/) (2025)

## Abstract (arXiv)

In the real world, medical datasets often exhibit a long-tailed data distribution (i.e., a few classes occupy most of the data, while most classes have rarely few samples), which results in a challenging imbalance learning scenario. For example, there are estimated more than 40 different kinds of retinal diseases with variable morbidity, however with more than 30+ conditions are very rare from the global patient cohorts, which results in a typical long-tailed learning problem for deep learning-based screening models. In this study, we propose class subset learning by dividing the long-tailed data into multiple class subsets according to prior knowledge, such as regions and phenotype information. It enforces the model to focus on learning the subset-specific knowledge. More specifically, there are some relational classes that reside in the fixed retinal regions, or some common pathological features are observed in both the majority and minority conditions. With those subsets learnt teacher models, then we are able to distill the multiple teacher models into a unified model with weighted knowledge distillation loss. The proposed framework proved to be effective for the long-tailed retinal diseases recognition task. The experimental results on two different datasets demonstrate that our method is flexible and can be easily plugged into many other state-of-the-art techniques with significant improvements.