# Metadata-enhanced contrastive learning from retinal optical coherence tomography images

arXiv: https://arxiv.org/abs/2208.02529 · HTML: https://mlsanity.com/paper/2208.02529/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 58 references that are in the map.
- First posted on arXiv 4 Aug 2022.

## Builds on (most cited first, 30 of 30)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [Exploring Simple Siamese Representation Learning](https://mlsanity.com/paper/2011.10566/) (2020)
- [Supervised Contrastive Learning](https://mlsanity.com/paper/2004.11362/) (2020)
- [Unsupervised Representation Learning by Predicting Image Rotations](https://mlsanity.com/paper/1803.07728/) (2018)
- [Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzles](https://mlsanity.com/paper/1603.09246/) (2016)
- [Big Self-Supervised Models are Strong Semi-Supervised Learners](https://mlsanity.com/paper/2006.10029/) (2020)
- [Colorful Image Colorization](https://mlsanity.com/paper/1603.08511/) (2016)
- [Dense Contrastive Learning for Self-Supervised Visual Pre-Training](https://mlsanity.com/paper/2011.09157/) (2020)
- [Debiased Contrastive Learning](https://mlsanity.com/paper/2007.00224/) (2020)
- [Big Self-Supervised Models Advance Medical Image Classification](https://mlsanity.com/paper/2101.05224/) (2021)
- [Contrastive learning of global and local features for medical image segmentation with limited annotations](https://mlsanity.com/paper/2006.10511/) (2020)
- [Self supervised contrastive learning for digital histopathology](https://mlsanity.com/paper/2011.13971/) (2020)
- [Self-Supervised Learning for Cardiac MR Image Segmentation by Anatomical Position Prediction](https://mlsanity.com/paper/1907.02757/) (2019)
- [Positional Contrastive Learning for Volumetric Medical Image Segmentation](https://mlsanity.com/paper/2106.09157/) (2021)
- [MedAug: Contrastive learning leveraging patient metadata improves representations for chest X-ray interpretation](https://mlsanity.com/paper/2102.10663/) (2021)
- [ContIG: Self-supervised Multimodal Contrastive Learning for Medical Imaging with Genetics](https://mlsanity.com/paper/2111.13424/) (2021)
- [USCL: Pretraining Deep Ultrasound Image Diagnosis Model through Video Contrastive Representation Learning](https://mlsanity.com/paper/2011.13066/) (2020)
- [Global and Local Contrastive Self-Supervised Learning for Semantic Segmentation of HR Remote Sensing Images](https://mlsanity.com/paper/2106.10605/) (2021)
- [Self-supervised Learning from 100 Million Medical Images](https://mlsanity.com/paper/2201.01283/) (2022)
- [Improving Cytoarchitectonic Segmentation of Human Brain Areas with Self-supervised Siamese Networks](https://mlsanity.com/paper/1806.05104/) (2018)
- [Modeling Disease Progression In Retinal OCTs With Longitudinal Self-Supervised Learning](https://mlsanity.com/paper/1910.09420/) (2019)
- [Dataset Growth in Medical Image Analysis Research](https://mlsanity.com/paper/1908.07765/) (2019)
- [Non-rigid image registration using fully convolutional networks with deep self-supervision](https://mlsanity.com/paper/1709.00799/) (2017)
- [On the Robustness of Pretraining and Self-Supervision for a Deep Learning-based Analysis of Diabetic Retinopathy](https://mlsanity.com/paper/2106.13497/) (2021)
- [Contrastive Learning with Temporal Correlated Medical Images: A Case Study using Lung Segmentation in Chest X-Rays](https://mlsanity.com/paper/2109.03233/) (2021)

## Built on by (most cited first, 14 of 14)

- [LaTiM: Longitudinal representation learning in continuous-time models to predict disease progression](https://mlsanity.com/paper/2404.07091/) (2024)
- [Tables Guide Vision: Learning to See the Heart through Tabular Data](https://mlsanity.com/paper/2503.14998/) (2025)
- [MR-CLIP: Efficient Metadata-Guided Learning of MRI Contrast Representations](https://mlsanity.com/paper/2507.00043/) (2025)
- [Clustering disease trajectories in contrastive feature space for biomarker discovery in age-related macular degeneration](https://mlsanity.com/paper/2301.04525/) (2023)
- [PRETI: Patient-Aware Retinal Foundation Model via Metadata-Guided Representation Learning](https://mlsanity.com/paper/2505.12233/) (2025)
- [Metadata-Aligned 3D MRI Representations for Contrast Understanding and Quality Control](https://mlsanity.com/paper/2511.00681/) (2025)
- [L-MAE: Longitudinal masked auto-encoder with time and severity-aware encoding for diabetic retinopathy progression prediction](https://mlsanity.com/paper/2403.16272/) (2024)
- [Deep-learning-based clustering of OCT images for biomarker discovery in age-related macular degeneration (Pinnacle study report 4)](https://mlsanity.com/paper/2405.09549/) (2024)
- [Learning Temporally Equivariance for Degenerative Disease Progression in OCT by Predicting Future Representations](https://mlsanity.com/paper/2405.09404/) (2024)
- [Morph-SSL: Self-Supervision with Longitudinal Morphing to Predict AMD Progression from OCT](https://mlsanity.com/paper/2304.08439/) (2023)
- [Chronological Contrastive Learning: Few-Shot Progression Assessment in Irreversible Diseases](https://mlsanity.com/paper/2603.21935/) (2026)
- [Weighting What Matters: Boosting Sample Efficiency in Medical Report Generation via Token Reweighting](https://mlsanity.com/paper/2604.21082/) (2026)
- [Representation learning from OCT images](https://mlsanity.com/paper/2605.02589/) (2026)
- [Who Needs Labels? Adapting Vision Foundation Models With the Metadata You Already Have](https://mlsanity.com/paper/2606.05107/) (2026)

## Abstract (arXiv)

Deep learning has potential to automate screening, monitoring and grading of disease in medical images. Pretraining with contrastive learning enables models to extract robust and generalisable features from natural image datasets, facilitating label-efficient downstream image analysis. However, the direct application of conventional contrastive methods to medical datasets introduces two domain-specific issues. Firstly, several image transformations which have been shown to be crucial for effective contrastive learning do not translate from the natural image to the medical image domain. Secondly, the assumption made by conventional methods, that any two images are dissimilar, is systematically misleading in medical datasets depicting the same anatomy and disease. This is exacerbated in longitudinal image datasets that repeatedly image the same patient cohort to monitor their disease progression over time. In this paper we tackle these issues by extending conventional contrastive frameworks with a novel metadata-enhanced strategy. Our approach employs widely available patient metadata to approximate the true set of inter-image contrastive relationships. To this end we employ records for patient identity, eye position (i.e. left or right) and time series information. In experiments using two large longitudinal datasets containing 170,427 retinal OCT images of 7,912 patients with age-related macular degeneration (AMD), we evaluate the utility of using metadata to incorporate the temporal dynamics of disease progression into pretraining. Our metadata-enhanced approach outperforms both standard contrastive methods and a retinal image foundation model in five out of six image-level downstream tasks related to AMD. Due to its modularity, our method can be quickly and cost-effectively tested to establish the potential benefits of including available metadata in contrastive pretraining.