# Multi-Modal Learning Using Physicians Diagnostics for Optical Coherence Tomography Classification

arXiv: https://arxiv.org/abs/2203.10622 · HTML: https://mlsanity.com/paper/2203.10622/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 6 of its 21 references that are in the map.
- First posted on arXiv 20 Mar 2022.

## Builds on (most cited first, 6 of 6)

- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Multimodal Machine Learning: A Survey and Taxonomy](https://mlsanity.com/paper/1705.09406/) (2017)
- [Positive-Congruent Training: Towards Regression-Free Model Updates](https://mlsanity.com/paper/2011.09161/) (2020)
- [Encoding Structure-Texture Relation with P-Net for Anomaly Detection in Retinal Images](https://mlsanity.com/paper/2008.03632/) (2020)
- [A Comparison of Pre-trained Vision-and-Language Models for Multimodal Representation Learning across Medical Images and Reports](https://mlsanity.com/paper/2009.01523/) (2020)
- [Unsupervised Multimodal Representation Learning across Medical Images and Reports](https://mlsanity.com/paper/1811.08615/) (2018)

## Built on by (most cited first, 7 of 7)

- [OLIVES Dataset: Ophthalmic Labels for Investigating Visual Eye Semantics](https://mlsanity.com/paper/2209.11195/) (2022)
- [Patient Aware Active Learning for Fine-Grained OCT Classification](https://mlsanity.com/paper/2206.11485/) (2022)
- [Probing the Purview of Neural Networks via Gradient Analysis](https://mlsanity.com/paper/2304.02834/) (2023)
- [Clinically Labeled Contrastive Learning for OCT Biomarker Classification](https://mlsanity.com/paper/2305.15154/) (2023)
- [A Large-scale Benchmark on Geological Fault Delineation Models: Domain Shift, Training Dynamics, Generalizability, Evaluation and Inferential Behavior](https://mlsanity.com/paper/2505.08585/) (2025)
- [Clinical Trial Active Learning](https://mlsanity.com/paper/2307.11209/) (2023)
- [TrajPRed: Trajectory Prediction with Region-based Relation Learning](https://mlsanity.com/paper/2404.06971/) (2024)

## Abstract (arXiv)

In this paper, we propose a framework that incorporates experts diagnostics and insights into the analysis of Optical Coherence Tomography (OCT) using multi-modal learning. To demonstrate the effectiveness of this approach, we create a medical diagnostic attribute dataset to improve disease classification using OCT. Although there have been successful attempts to deploy machine learning for disease classification in OCT, such methodologies lack the experts insights. We argue that injecting ophthalmological assessments as another supervision in a learning framework is of great importance for the machine learning process to perform accurate and interpretable classification. We demonstrate the proposed framework through comprehensive experiments that compare the effectiveness of combining diagnostic attribute features with latent visual representations and show that they surpass the state-of-the-art approach. Finally, we analyze the proposed dual-stream architecture and provide an insight that determine the components that contribute most to classification performance.