# MultiEYE: Dataset and Benchmark for OCT-Enhanced Retinal Disease Recognition from Fundus Images

arXiv: https://arxiv.org/abs/2412.09402 · HTML: https://mlsanity.com/paper/2412.09402/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 57 references that are in the map.
- First posted on arXiv 12 Dec 2024.

## Builds on (most cited first, 30 of 33)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Visual Classification via Description from Large Language Models](https://mlsanity.com/paper/2210.07183/) (2022)
- [Language in a Bottle: Language Model Guided Concept Bottlenecks for Interpretable Image Classification](https://mlsanity.com/paper/2211.11158/) (2022)
- [Label-Free Concept Bottleneck Models](https://mlsanity.com/paper/2304.06129/) (2023)
- [What does a platypus look like? Generating customized prompts for zero-shot image classification](https://mlsanity.com/paper/2209.03320/) (2022)
- [Learning Deep Representations with Probabilistic Knowledge Transfer](https://mlsanity.com/paper/1803.10837/) (2018)
- [Cross Modal Distillation for Supervision Transfer](https://mlsanity.com/paper/1507.00448/) (2015)
- [A Foundation Language-Image Model of the Retina (FLAIR): Encoding Expert Knowledge in Text Supervision](https://mlsanity.com/paper/2308.07898/) (2023)
- [Holistic Autonomous Driving Understanding by Bird's-Eye-View Injected Multi-Modal Large Models](https://mlsanity.com/paper/2401.00988/) (2024)
- [Distilling Audio-Visual Knowledge by Compositional Contrastive Learning](https://mlsanity.com/paper/2104.10955/) (2021)
- [Robust and Interpretable Medical Image Classifiers via Concept Bottleneck Models](https://mlsanity.com/paper/2310.03182/) (2023)
- [OCTID: Optical Coherence Tomography Image Database](https://mlsanity.com/paper/1812.07056/) (2018)
- [A Joint Cross-Attention Model for Audio-Visual Fusion in Dimensional Emotion Recognition](https://mlsanity.com/paper/2203.14779/) (2022)
- [A ChatGPT Aided Explainable Framework for Zero-Shot Medical Image Diagnosis](https://mlsanity.com/paper/2307.01981/) (2023)
- [Interpretable Bilingual Multimodal Large Language Model for Diverse Biomedical Tasks](https://mlsanity.com/paper/2410.18387/) (2024)
- [Concept Bottleneck with Visual Concept Filtering for Explainable Medical Image Classification](https://mlsanity.com/paper/2308.11920/) (2023)
- [Cross Attentional Audio-Visual Fusion for Dimensional Emotion Recognition](https://mlsanity.com/paper/2111.05222/) (2021)
- [Learning Two-Stream CNN for Multi-Modal Age-related Macular Degeneration Categorization](https://mlsanity.com/paper/2012.01879/) (2020)
- [Multi-Modal Multi-Instance Learning for Retinal Disease Recognition](https://mlsanity.com/paper/2109.12307/) (2021)
- [Dataset and Evaluation algorithm design for GOALS Challenge](https://mlsanity.com/paper/2207.14447/) (2022)
- [Two-Stream CNN with Loose Pair Training for Multi-modal AMD Categorization](https://mlsanity.com/paper/1907.12023/) (2019)
- [Multimodal Information Fusion for Glaucoma and DR Classification](https://mlsanity.com/paper/2209.00979/) (2022)
- [LLMs as Visual Explainers: Advancing Image Classification with Evolving Visual Descriptions](https://mlsanity.com/paper/2311.11904/) (2023)
- [FoPro-KD: Fourier Prompted Effective Knowledge Distillation for Long-Tailed Medical Image Recognition](https://mlsanity.com/paper/2305.17421/) (2023)
- [Free Lunch for Surgical Video Understanding by Distilling Self-Supervisions](https://mlsanity.com/paper/2205.09292/) (2022)
- [Text Descriptions are Compressive and Invariant Representations for Visual Learning](https://mlsanity.com/paper/2307.04317/) (2023)

## Built on by (most cited first, 9 of 9)

- [PGAD: Prototype-Guided Adaptive Distillation for Multi-Modal Learning in AD Diagnosis](https://mlsanity.com/paper/2503.04836/) (2025)
- [UOPSL: Unpaired OCT Predilection Sites Learning for Fundus Image Diagnosis Augmentation](https://mlsanity.com/paper/2509.08624/) (2025)
- [Forgetting-Resistant and Lesion-Aware Source-Free Domain Adaptive Fundus Image Analysis with Vision-Language Model](https://mlsanity.com/paper/2602.19471/) (2026)
- [MedForge: Interpretable Medical Deepfake Detection via Forgery-aware Reasoning](https://mlsanity.com/paper/2603.18577/) (2026)
- [Representation learning from OCT images](https://mlsanity.com/paper/2605.02589/) (2026)
- [EyeMVP: OCT-Informed Fundus Representation Learning via Paired CFP--OCT Pretraining](https://mlsanity.com/paper/2606.15129/) (2026)
- [Cross-Modal Fusion of OCT and OCT angiography enface for Improved Diagnostics of Diabetic Retinopathy](https://mlsanity.com/paper/2607.03959/) (2026)
- [Shared Semantic Codebook Distillation for Unpaired Cross-Modal Medical Classification](https://mlsanity.com/paper/2607.27357/) (2026)
- [FOCUS: Benchmarking Retinal Model Generalization from Foundation Vision Encoders to Multimodal LLMs](https://mlsanity.com/paper/2609.33158/) (2026)

## Abstract (arXiv)

Existing multi-modal learning methods on fundus and OCT images mostly require both modalities to be available and strictly paired for training and testing, which appears less practical in clinical scenarios. To expand the scope of clinical applications, we formulate a novel setting, "OCT-enhanced disease recognition from fundus images", that allows for the use of unpaired multi-modal data during the training phase and relies on the widespread fundus photographs for testing. To benchmark this setting, we present the first large multi-modal multi-class dataset for eye disease diagnosis, MultiEYE, and propose an OCT-assisted Conceptual Distillation Approach (OCT-CoDA), which employs semantically rich concepts to extract disease-related knowledge from OCT images and leverage them into the fundus model. Specifically, we regard the image-concept relation as a link to distill useful knowledge from the OCT teacher model to the fundus student model, which considerably improves the diagnostic performance based on fundus images and formulates the cross-modal knowledge transfer into an explainable process. Through extensive experiments on the multi-disease classification task, our proposed OCT-CoDA demonstrates remarkable results and interpretability, showing great potential for clinical application. Our dataset and code are available at https://github.com/xmed-lab/MultiEYE.