# OLIVES Dataset: Ophthalmic Labels for Investigating Visual Eye Semantics

arXiv: https://arxiv.org/abs/2209.11195 · HTML: https://mlsanity.com/paper/2209.11195/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 86 references that are in the map.
- First posted on arXiv 22 Sep 2022.

## Builds on (most cited first, 29 of 29)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [Supervised Contrastive Learning](https://mlsanity.com/paper/2004.11362/) (2020)
- [Improved Baselines with Momentum Contrastive Learning](https://mlsanity.com/paper/2003.04297/) (2020)
- [Prototypical Contrastive Learning of Unsupervised Representations](https://mlsanity.com/paper/2005.04966/) (2020)
- [The Future of Digital Health with Federated Learning](https://mlsanity.com/paper/2003.08119/) (2020)
- [Keystroke Dynamics as Part of Lifelogging](https://mlsanity.com/paper/2010.10144/) (2020)
- [Ethical Machine Learning in Health Care](https://mlsanity.com/paper/2009.10576/) (2020)
- [Attention Based Glaucoma Detection: A Large-scale Database and CNN Model](https://mlsanity.com/paper/1903.10831/) (2019)
- [Visual Causal Feature Learning](https://mlsanity.com/paper/1412.2309/) (2014)
- [A Benchmark of Ocular Disease Intelligent Recognition: One Shot for Multi-disease Detection](https://mlsanity.com/paper/2102.07978/) (2021)
- [CURE-TSR: Challenging Unreal and Real Environments for Traffic Sign Recognition](https://mlsanity.com/paper/1712.02463/) (2017)
- [A Machine Learning Benchmark for Facies Classification](https://mlsanity.com/paper/1901.07659/) (2019)
- [Contrastive Explanations in Neural Networks](https://mlsanity.com/paper/2008.00178/) (2020)
- [Modeling Disease Progression In Retinal OCTs With Longitudinal Self-Supervised Learning](https://mlsanity.com/paper/1910.09420/) (2019)
- [Action Segmentation with Mixed Temporal Domain Adaptation](https://mlsanity.com/paper/2104.07461/) (2021)
- [CURE-OR: Challenging Unreal and Real Environments for Object Recognition](https://mlsanity.com/paper/1810.08293/) (2018)
- [A feature agnostic approach for glaucoma detection in OCT volumes](https://mlsanity.com/paper/1807.04855/) (2018)
- [Extracting Causal Visual Features for Limited label Classification](https://mlsanity.com/paper/2103.12322/) (2021)
- [Volumetric Supervised Contrastive Learning for Seismic Semantic Segmentation](https://mlsanity.com/paper/2206.08158/) (2022)
- [Contrastive Reasoning in Neural Networks](https://mlsanity.com/paper/2103.12329/) (2021)
- [Patient Aware Active Learning for Fine-Grained OCT Classification](https://mlsanity.com/paper/2206.11485/) (2022)
- [Relative Afferent Pupillary Defect Screening through Transfer Learning](https://mlsanity.com/paper/1908.02300/) (2019)
- [Multi-Modal Learning Using Physicians Diagnostics for Optical Coherence Tomography Classification](https://mlsanity.com/paper/2203.10622/) (2022)
- [DECAL: DEployable Clinical Active Learning](https://mlsanity.com/paper/2206.10120/) (2022)
- [Fabric Surface Characterization: Assessment of Deep Learning-based Texture Representations Using a Challenging Dataset](https://mlsanity.com/paper/2003.07725/) (2020)

## Built on by (most cited first, 24 of 24)

- [OmniMedVQA: A New Large-Scale Comprehensive Evaluation Benchmark for Medical LVLM](https://mlsanity.com/paper/2402.09181/) (2024)
- [A review of deep learning-based information fusion techniques for multimodal medical image classification](https://mlsanity.com/paper/2404.15022/) (2024)
- [Clinically Labeled Contrastive Learning for OCT Biomarker Classification](https://mlsanity.com/paper/2305.15154/) (2023)
- [Improving Representation of High-frequency Components for Medical Visual Foundation Models](https://mlsanity.com/paper/2407.14651/) (2024)
- [Hierarchical and Multimodal Data for Daily Activity Understanding](https://mlsanity.com/paper/2504.17696/) (2025)
- [MIRAGE: Multimodal foundation model and benchmark for comprehensive retinal OCT image analysis](https://mlsanity.com/paper/2506.08900/) (2025)
- [Confidence-aware multi-modality learning for eye disease screening](https://mlsanity.com/paper/2405.18167/) (2024)
- [F3OCUS -- Federated Finetuning of Vision-Language Foundation Models with Optimal Client Layer Updating Strategy via Multi-objective Meta-Heuristics](https://mlsanity.com/paper/2411.11912/) (2024)
- [A Large-scale Benchmark on Geological Fault Delineation Models: Domain Shift, Training Dynamics, Generalizability, Evaluation and Inferential Behavior](https://mlsanity.com/paper/2505.08585/) (2025)
- [On the Ramifications of Human Label Uncertainty](https://mlsanity.com/paper/2211.05871/) (2022)
- [Clinical Trial Active Learning](https://mlsanity.com/paper/2307.11209/) (2023)
- [Taxes Are All You Need: Integration of Taxonomical Hierarchy Relationships into the Contrastive Loss](https://mlsanity.com/paper/2406.06848/) (2024)
- [Clinical Contrastive Learning for Biomarker Detection](https://mlsanity.com/paper/2211.05092/) (2022)
- [Stochastic Surprisal: An inferential measurement of Free Energy in Neural Networks](https://mlsanity.com/paper/2302.05776/) (2023)
- [Transformer-based end-to-end classification of variable-length volumetric data](https://mlsanity.com/paper/2307.06666/) (2023)
- [Are Objective Explanatory Evaluation metrics Trustworthy? An Adversarial Analysis](https://mlsanity.com/paper/2406.07820/) (2024)
- [CRACKS: Crowdsourcing Resources for Analysis and Categorization of Key Subsurface faults](https://mlsanity.com/paper/2408.11185/) (2024)
- [Project Imaging-X: A Survey of 1000+ Open-Access Medical Imaging Datasets for Foundation Model Development](https://mlsanity.com/paper/2603.27460/) (2026)
- [Visual Question Answering in Ophthalmology: A Progressive and Practical Perspective](https://mlsanity.com/paper/2410.16662/) (2024)
- [Countering Multi-modal Representation Collapse through Rank-targeted Fusion](https://mlsanity.com/paper/2511.06450/) (2025)
- [Leveraging Complementary Attention maps in vision transformers for OCT image analysis](https://mlsanity.com/paper/2310.14005/) (2023)
- [Code and Pixels: Multi-Modal Contrastive Pre-training for Enhanced Tabular Data Analysis](https://mlsanity.com/paper/2501.07304/) (2025)
- [Automated 2D and 3D Segmentation of AMD and DME Lesions in OCT](https://mlsanity.com/paper/2608.27095/) (2026)
- [SynCo: Learning Cross-Modal Synergy by Contrasting Interaction Residuals](https://mlsanity.com/paper/2609.32846/) (2026)

## Abstract (arXiv)

Clinical diagnosis of the eye is performed over multifarious data modalities including scalar clinical labels, vectorized biomarkers, two-dimensional fundus images, and three-dimensional Optical Coherence Tomography (OCT) scans. Clinical practitioners use all available data modalities for diagnosing and treating eye diseases like Diabetic Retinopathy (DR) or Diabetic Macular Edema (DME). Enabling usage of machine learning algorithms within the ophthalmic medical domain requires research into the relationships and interactions between all relevant data over a treatment period. Existing datasets are limited in that they neither provide data nor consider the explicit relationship modeling between the data modalities. In this paper, we introduce the Ophthalmic Labels for Investigating Visual Eye Semantics (OLIVES) dataset that addresses the above limitation. This is the first OCT and near-IR fundus dataset that includes clinical labels, biomarker labels, disease labels, and time-series patient treatment information from associated clinical trials. The dataset consists of 1268 near-IR fundus images each with at least 49 OCT scans, and 16 biomarkers, along with 4 clinical labels and a disease diagnosis of DR or DME. In total, there are 96 eyes' data averaged over a period of at least two years with each eye treated for an average of 66 weeks and 7 injections. We benchmark the utility of OLIVES dataset for ophthalmic data as well as provide benchmarks and concrete research directions for core and emerging machine learning paradigms within medical image analysis.