# Multi-Modal Multi-Instance Learning for Retinal Disease Recognition

arXiv: https://arxiv.org/abs/2109.12307 · HTML: https://mlsanity.com/paper/2109.12307/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 25 references that are in the map.
- First posted on arXiv 25 Sep 2021.

## Builds on (most cited first, 13 of 13)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [Attention-based Deep Multiple Instance Learning](https://mlsanity.com/paper/1802.04712/) (2018)
- [Transformer Interpretability Beyond Attention Visualization](https://mlsanity.com/paper/2012.09838/) (2020)
- [Can Spatiotemporal 3D CNNs Retrace the History of 2D CNNs and ImageNet?](https://mlsanity.com/paper/1711.09577/) (2017)
- [Super-Convergence: Very Fast Training of Neural Networks Using Large Learning Rates](https://mlsanity.com/paper/1708.07120/) (2017)
- [Look, Listen, and Attend: Co-Attention Network for Self-Supervised Audio-Visual Representation Learning](https://mlsanity.com/paper/2008.05789/) (2020)
- [Two-Stream CNN with Loose Pair Training for Multi-modal AMD Categorization](https://mlsanity.com/paper/1907.12023/) (2019)
- [Deep learning is effective for the classification of OCT images of normal versus Age-related Macular Degeneration](https://mlsanity.com/paper/1612.04891/) (2016)
- [Evaluating Two-Stream CNN for Video Classification](https://mlsanity.com/paper/1504.01920/) (2015)

## Built on by (most cited first, 14 of 14)

- [MM-Retinal: Knowledge-Enhanced Foundational Pretraining with Fundus Image-Text Expertise](https://mlsanity.com/paper/2405.11793/) (2024)
- [Learn to Understand Negation in Video Retrieval](https://mlsanity.com/paper/2205.00132/) (2022)
- [MultiEYE: Dataset and Benchmark for OCT-Enhanced Retinal Disease Recognition from Fundus Images](https://mlsanity.com/paper/2412.09402/) (2024)
- [Detecting Heart Disease from Multi-View Ultrasound Images via Supervised Attention Multiple Instance Learning](https://mlsanity.com/paper/2306.00003/) (2023)
- [Fundus-Enhanced Disease-Aware Distillation Model for Retinal Disease Classification from OCT Images](https://mlsanity.com/paper/2308.00291/) (2023)
- [An Organism Starts with a Single Pix-Cell: A Neural Cellular Diffusion for High-Resolution Image Synthesis](https://mlsanity.com/paper/2407.03018/) (2024)
- [A Survey of Multimodal Ophthalmic Diagnostics: From Task-Specific Approaches to Foundational Models](https://mlsanity.com/paper/2508.03734/) (2025)
- [Supervised Domain Adaptation for Recognizing Retinal Diseases from Wide-Field Fundus Images](https://mlsanity.com/paper/2305.08078/) (2023)
- [ETSCL: An Evidence Theory-Based Supervised Contrastive Learning Framework for Multi-modal Glaucoma Grading](https://mlsanity.com/paper/2407.14230/) (2024)
- [Cross-Fundus Transformer for Multi-modal Diabetic Retinopathy Grading with Cataract](https://mlsanity.com/paper/2411.00726/) (2024)
- [Lesion Localization in OCT by Semi-Supervised Object Detection](https://mlsanity.com/paper/2204.11227/) (2022)
- [Adaptive Fusion of Radiomics and Deep Features for Lung Adenocarcinoma Subtype Recognition](https://mlsanity.com/paper/2308.13997/) (2023)
- [Multiscale Color Guided Attention Ensemble Classifier for Age-Related Macular Degeneration using Concurrent Fundus and Optical Coherence Tomography Images](https://mlsanity.com/paper/2409.00718/) (2024)
- [Early Intervention for VFM-based Multimodal Medical Image Classification](https://mlsanity.com/paper/2603.17514/) (2026)

## Abstract (arXiv)

This paper attacks an emerging challenge of multi-modal retinal disease recognition. Given a multi-modal case consisting of a color fundus photo (CFP) and an array of OCT B-scan images acquired during an eye examination, we aim to build a deep neural network that recognizes multiple vision-threatening diseases for the given case. As the diagnostic efficacy of CFP and OCT is disease-dependent, the network's ability of being both selective and interpretable is important. Moreover, as both data acquisition and manual labeling are extremely expensive in the medical domain, the network has to be relatively lightweight for learning from a limited set of labeled multi-modal samples. Prior art on retinal disease recognition focuses either on a single disease or on a single modality, leaving multi-modal fusion largely underexplored. We propose in this paper Multi-Modal Multi-Instance Learning (MM-MIL) for selectively fusing CFP and OCT modalities. Its lightweight architecture (as compared to current multi-head attention modules) makes it suited for learning from relatively small-sized datasets. For an effective use of MM-MIL, we propose to generate a pseudo sequence of CFPs by over sampling a given CFP. The benefits of this tactic include well balancing instances across modalities, increasing the resolution of the CFP input, and finding out regions of the CFP most relevant with respect to the final diagnosis. Extensive experiments on a real-world dataset consisting of 1,206 multi-modal cases from 1,193 eyes of 836 subjects demonstrate the viability of the proposed model.