# Masked Image Modelling for retinal OCT understanding

arXiv: https://arxiv.org/abs/2405.14788 · HTML: https://mlsanity.com/paper/2405.14788/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 34 references that are in the map.
- First posted on arXiv 23 May 2024.

## Builds on (most cited first, 13 of 13)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Exploring Plain Vision Transformer Backbones for Object Detection](https://mlsanity.com/paper/2203.16527/) (2022)
- [Are Natural Domain Foundation Models Useful for Medical Image Classification?](https://mlsanity.com/paper/2310.19522/) (2023)
- [OCTID: Optical Coherence Tomography Image Database](https://mlsanity.com/paper/1812.07056/) (2018)
- [Pretrained ViTs Yield Versatile Representations For Medical Images](https://mlsanity.com/paper/2303.07034/) (2023)
- [TINC: Temporally Informed Non-Contrastive Learning for Disease Progression Modeling in Retinal OCT Volumes](https://mlsanity.com/paper/2206.15282/) (2022)
- [Two-Stream CNN with Loose Pair Training for Multi-modal AMD Categorization](https://mlsanity.com/paper/1907.12023/) (2019)
- [Multimodal Information Fusion for Glaucoma and DR Classification](https://mlsanity.com/paper/2209.00979/) (2022)
- [Pathological OCT Retinal Layer Segmentation using Branch Residual U-shape Networks](https://mlsanity.com/paper/1707.04931/) (2017)
- [DISCOVER: 2-D Multiview Summarization of Optical Coherence Tomography Angiography for Automatic Diabetic Retinopathy Diagnosis](https://mlsanity.com/paper/2401.05137/) (2024)
- [Deep Multimodal Fusion of Data with Heterogeneous Dimensionality via Projective Networks](https://mlsanity.com/paper/2402.01311/) (2024)

## Built on by (most cited first, 2 of 2)

- [VAMAE: Vessel-Aware Masked Autoencoders for OCT Angiography](https://mlsanity.com/paper/2604.06583/) (2026)
- [M-IDoL: Information Decomposition for Modality-Specific and Diverse Representation Learning in Medical Foundation Model](https://mlsanity.com/paper/2604.08936/) (2026)

## Abstract (arXiv)

This work explores the effectiveness of masked image modelling for learning representations of retinal OCT images. To this end, we leverage Masked Autoencoders (MAE), a simple and scalable method for self-supervised learning, to obtain a powerful and general representation for OCT images by training on 700K OCT images from 41K patients collected under real world clinical settings. We also provide the first extensive evaluation for a model of OCT on a challenging battery of 6 downstream tasks. Our model achieves strong performance when fully finetuned but can also serve as a versatile frozen feature extractor for many tasks using lightweight adapters. Furthermore, we propose an extension of the MAE pretraining to fuse OCT with an auxiliary modality, namely, IR fundus images and learn a joint model for both. We demonstrate our approach improves performance on a multimodal downstream application. Our experiments utilize most publicly available OCT datasets, thus enabling future comparisons. Our code and model weights are publicly available https://github.com/TheoPis/MIM_OCT.