# SSiT: Saliency-guided Self-supervised Image Transformer for Diabetic Retinopathy Grading

arXiv: https://arxiv.org/abs/2210.10969 · HTML: https://mlsanity.com/paper/2210.10969/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 42 of its 77 references that are in the map.
- First posted on arXiv 20 Oct 2022.

## Builds on (most cited first, 30 of 42)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [Unsupervised Learning of Visual Features by Contrasting Cluster Assignments](https://mlsanity.com/paper/2006.09882/) (2020)
- [BEiT: BERT Pre-Training of Image Transformers](https://mlsanity.com/paper/2106.08254/) (2021)
- [Improved Baselines with Momentum Contrastive Learning](https://mlsanity.com/paper/2003.04297/) (2020)
- [Unsupervised Feature Learning via Non-Parametric Instance-level Discrimination](https://mlsanity.com/paper/1805.01978/) (2018)
- [An Empirical Study of Training Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.02057/) (2021)
- [Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour](https://mlsanity.com/paper/1706.02677/) (2017)
- [SimMIM: A Simple Framework for Masked Image Modeling](https://mlsanity.com/paper/2111.09886/) (2021)
- [UNETR: Transformers for 3D Medical Image Segmentation](https://mlsanity.com/paper/2103.10504/) (2021)
- [Big Transfer (BiT): General Visual Representation Learning](https://mlsanity.com/paper/1912.11370/) (2019)
- [Dense Contrastive Learning for Self-Supervised Visual Pre-Training](https://mlsanity.com/paper/2011.09157/) (2020)
- [Self-Supervised Pre-Training of Swin Transformers for 3D Medical Image Analysis](https://mlsanity.com/paper/2111.14791/) (2021)
- [Big Self-Supervised Models Advance Medical Image Classification](https://mlsanity.com/paper/2101.05224/) (2021)
- [Contrastive learning of global and local features for medical image segmentation with limited annotations](https://mlsanity.com/paper/2006.10511/) (2020)
- [Multi-Granularity Cross-modal Alignment for Generalized Medical Visual Representation Learning](https://mlsanity.com/paper/2210.06044/) (2022)
- [Efficient Self-supervised Vision Transformers for Representation Learning](https://mlsanity.com/paper/2106.09785/) (2021)
- [MST: Masked Self-Supervised Transformer for Visual Representation](https://mlsanity.com/paper/2106.05656/) (2021)
- [Self-supervised Feature Learning for 3D Medical Images by Playing a Rubik's Cube](https://mlsanity.com/paper/1910.02241/) (2019)
- [Is it Time to Replace CNNs with Transformers for Medical Images?](https://mlsanity.com/paper/2108.09038/) (2021)
- [Preservational Learning Improves Self-supervised Medical Image Models by Reconstructing Diverse Contexts](https://mlsanity.com/paper/2109.04379/) (2021)

## Built on by (most cited first, 9 of 9)

- [Pathology-Aware Prototype Evolution via LLM-Driven Semantic Disambiguation for Multicenter Diabetic Retinopathy Diagnosis](https://mlsanity.com/paper/2511.22033/) (2025)
- [Fine-grained Prompt Tuning: A Parameter and Memory Efficient Transfer Learning Method for High-resolution Medical Image Classification](https://mlsanity.com/paper/2403.07576/) (2024)
- [Text-guided Foundation Model Adaptation for Long-Tailed Medical Image Classification](https://mlsanity.com/paper/2408.14770/) (2024)
- [Detecting Severity of Diabetic Retinopathy from Fundus Images: A Transformer Network-based Review](https://mlsanity.com/paper/2301.00973/) (2023)
- [Boosting Memory Efficiency in Transfer Learning for High-Resolution Medical Image Classification](https://mlsanity.com/paper/2408.02426/) (2024)
- [Explainable Image Quality Assessment for Medical Imaging](https://mlsanity.com/paper/2303.14479/) (2023)
- [Saliency-guided and Patch-based Mixup for Long-tailed Skin Cancer Image Classification](https://mlsanity.com/paper/2406.10801/) (2024)
- [Domain Adaptive Diabetic Retinopathy Grading with Model Absence and Flowing Data](https://mlsanity.com/paper/2412.01203/) (2024)
- [Learning to Fuse and Reconstruct Multi-View Graphs for Diabetic Retinopathy Grading](https://mlsanity.com/paper/2602.21944/) (2026)

## Abstract (arXiv)

Self-supervised Learning (SSL) has been widely applied to learn image representations through exploiting unlabeled images. However, it has not been fully explored in the medical image analysis field. In this work, Saliency-guided Self-Supervised image Transformer (SSiT) is proposed for Diabetic Retinopathy (DR) grading from fundus images. We novelly introduce saliency maps into SSL, with a goal of guiding self-supervised pre-training with domain-specific prior knowledge. Specifically, two saliency-guided learning tasks are employed in SSiT: (1) Saliency-guided contrastive learning is conducted based on the momentum contrast, wherein fundus images' saliency maps are utilized to remove trivial patches from the input sequences of the momentum-updated key encoder. Thus, the key encoder is constrained to provide target representations focusing on salient regions, guiding the query encoder to capture salient features. (2) The query encoder is trained to predict the saliency segmentation, encouraging the preservation of fine-grained information in the learned representations. To assess our proposed method, four publicly-accessible fundus image datasets are adopted. One dataset is employed for pre-training, while the three others are used to evaluate the pre-trained models' performance on downstream DR grading. The proposed SSiT significantly outperforms other representative state-of-the-art SSL methods on all downstream datasets and under various evaluation settings. For example, SSiT achieves a Kappa score of 81.88% on the DDR dataset under fine-tuning evaluation, outperforming all other ViT-based SSL methods by at least 9.48%.