# OCT-SelfNet: A Self-Supervised Framework with Multi-Modal Datasets for Generalized and Robust Retinal Disease Detection

arXiv: https://arxiv.org/abs/2401.12344 · HTML: https://mlsanity.com/paper/2401.12344/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 55 references that are in the map.
- First posted on arXiv 22 Jan 2024.

## Builds on (most cited first, 23 of 23)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Going Deeper with Convolutions](https://mlsanity.com/paper/1409.4842/) (2014)
- [BEiT: BERT Pre-Training of Image Transformers](https://mlsanity.com/paper/2106.08254/) (2021)
- [Swin Transformer V2: Scaling Up Capacity and Resolution](https://mlsanity.com/paper/2111.09883/) (2021)
- [SimMIM: A Simple Framework for Masked Image Modeling](https://mlsanity.com/paper/2111.09886/) (2021)
- [Self-supervised Visual Feature Learning with Deep Neural Networks: A Survey](https://mlsanity.com/paper/1902.06162/) (2019)
- [Transfer Learning for Domain Adaptation in MRI: Application in Brain Lesion Segmentation](https://mlsanity.com/paper/1702.07841/) (2017)
- [OCTA-500: A Retinal Dataset for Optical Coherence Tomography Angiography Study](https://mlsanity.com/paper/2012.07261/) (2020)
- [A survey on efficient vision transformers: algorithms, techniques, and performance benchmarking](https://mlsanity.com/paper/2309.02031/) (2023)
- [Comparing Different Deep Learning Architectures for Classification of Chest Radiographs](https://mlsanity.com/paper/2002.08991/) (2020)
- [Recent Advances in Vision Transformer: A Survey and Outlook of Recent Work](https://mlsanity.com/paper/2203.01536/) (2022)
- [Multi-Scale Convolutional Neural Network for Automated AMD Classification using Retinal OCT Images](https://mlsanity.com/paper/2110.03002/) (2021)
- [Transfer Learning for Automated OCTA Detection of Diabetic Retinopathy](https://mlsanity.com/paper/1910.01796/) (2019)
- [AV-Net: Deep learning for fully automated artery-vein classification in optical coherence tomography angiography](https://mlsanity.com/paper/2006.03742/) (2020)
- [Deep learning is effective for the classification of OCT images of normal versus Age-related Macular Degeneration](https://mlsanity.com/paper/1612.04891/) (2016)

## Built on by (most cited first, 2 of 2)

- [Multi-OCT-SelfNet: Integrating Self-Supervised Learning with Multi-Source Data Fusion for Enhanced Multi-Class Retinal Disease Classification](https://mlsanity.com/paper/2409.11375/) (2024)
- [RetinaVision: XAI-Driven Augmented Regulation for Precise Retinal Disease Classification using deep learning framework](https://mlsanity.com/paper/2602.19324/) (2026)

## Abstract (arXiv)

Despite the revolutionary impact of AI and the development of locally trained algorithms, achieving widespread generalized learning from multi-modal data in medical AI remains a significant challenge. This gap hinders the practical deployment of scalable medical AI solutions. Addressing this challenge, our research contributes a self-supervised robust machine learning framework, OCT-SelfNet, for detecting eye diseases using optical coherence tomography (OCT) images. In this work, various data sets from various institutions are combined enabling a more comprehensive range of representation. Our method addresses the issue using a two-phase training approach that combines self-supervised pretraining and supervised fine-tuning with a mask autoencoder based on the SwinV2 backbone by providing a solution for real-world clinical deployment. Extensive experiments on three datasets with different encoder backbones, low data settings, unseen data settings, and the effect of augmentation show that our method outperforms the baseline model, Resnet-50 by consistently attaining AUC-ROC performance surpassing 77% across all tests, whereas the baseline model exceeds 54%. Moreover, in terms of the AUC-PR metric, our proposed method exceeded 42%, showcasing a substantial increase of at least 10% in performance compared to the baseline, which exceeded only 33%. This contributes to our understanding of our approach's potential and emphasizes its usefulness in clinical settings.