# Local-to-Global Self-Supervised Representation Learning for Diabetic Retinopathy Grading

arXiv: https://arxiv.org/abs/2410.00779 · HTML: https://mlsanity.com/paper/2410.00779/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Distillation for medical imaging, one of 12 directions in Knowledge distillation.
- Builds on 18 of its 35 references that are in the map.
- First posted on arXiv 1 Oct 2024.

## Placed in

- Knowledge distillation / [Distillation for medical imaging](https://mlsanity.com/distill/distillation-for-medical-imaging/)

## Builds on (most cited first, 18 of 18)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Unsupervised Learning of Visual Features by Contrasting Cluster Assignments](https://mlsanity.com/paper/2006.09882/) (2020)
- [Learning Deep Features for Discriminative Localization](https://mlsanity.com/paper/1512.04150/) (2015)
- [iBOT: Image BERT Pre-Training with Online Tokenizer](https://mlsanity.com/paper/2111.07832/) (2021)
- [Transformers in Medical Imaging: A Survey](https://mlsanity.com/paper/2201.09873/) (2022)
- [Large scale distributed neural network training through online distillation](https://mlsanity.com/paper/1804.03235/) (2018)
- [Knowledge Distillation Meets Self-Supervision](https://mlsanity.com/paper/2006.07114/) (2020)
- [Boosting Self-Supervised Learning via Knowledge Transfer](https://mlsanity.com/paper/1805.00385/) (2018)
- [Teaching Matters: Investigating the Role of Supervision in Vision Transformers](https://mlsanity.com/paper/2212.03862/) (2022)
- [Knowledge Distillation in Deep Learning and its Applications](https://mlsanity.com/paper/2007.09029/) (2020)
- [S2-BNN: Bridging the Gap Between Self-Supervised Real and 1-bit Neural Networks via Guided Distribution Calibration](https://mlsanity.com/paper/2102.08946/) (2021)
- [Predicting Risk of Developing Diabetic Retinopathy using Deep Learning](https://mlsanity.com/paper/2008.04370/) (2020)
- [Contrastive learning-based pretraining improves representation and transferability of diabetic retinopathy classification models](https://mlsanity.com/paper/2208.11563/) (2022)
- [Semi-weakly Supervised Contrastive Representation Learning for Retinal Fundus Images](https://mlsanity.com/paper/2108.02122/) (2021)
- [Self-Supervised Learning from Unlabeled Fundus Photographs Improves Segmentation of the Retina](https://mlsanity.com/paper/2108.02798/) (2021)

## Abstract (arXiv)

Artificial intelligence algorithms have demonstrated their image classification and segmentation ability in the past decade. However, artificial intelligence algorithms perform less for actual clinical data than those used for simulations. This research aims to present a novel hybrid learning model using self-supervised learning and knowledge distillation, which can achieve sufficient generalization and robustness. The self-attention mechanism and tokens employed in ViT, besides the local-to-global learning approach used in the hybrid model, enable the proposed algorithm to extract a high-dimensional and high-quality feature space from images. To demonstrate the proposed neural network's capability in classifying and extracting feature spaces from medical images, we use it on a dataset of Diabetic Retinopathy images, specifically the EyePACS dataset. This dataset is more complex structurally and challenging regarding damaged areas than other medical images. For the first time in this study, self-supervised learning and knowledge distillation are used to classify this dataset. In our algorithm, for the first time among all self-supervised learning and knowledge distillation models, the test dataset is 50% larger than the training dataset. Unlike many studies, we have not removed any images from the dataset. Finally, our algorithm achieved an accuracy of 79.1% in the linear classifier and 74.36% in the k-NN algorithm for multiclass classification. Compared to a similar state-of-the-art model, our results achieved higher accuracy and more effective representation spaces.