# Leveraging Multimodal Fusion for Enhanced Diagnosis of Multiple Retinal Diseases in Ultra-wide OCTA

arXiv: https://arxiv.org/abs/2311.10331 · HTML: https://mlsanity.com/paper/2311.10331/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 13 of its 24 references that are in the map.
- First posted on arXiv 17 Nov 2023.

## Builds on (most cited first, 13 of 13)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [A ConvNet for the 2020s](https://mlsanity.com/paper/2201.03545/) (2022)
- [MultiMAE: Multi-modal Multi-task Masked Autoencoders](https://mlsanity.com/paper/2204.01678/) (2022)
- [ROSE: A Retinal OCT-Angiography Vessel Segmentation Dataset and New Model](https://mlsanity.com/paper/2007.05201/) (2020)
- [Generalist Vision Foundation Models for Medical Imaging: A Case Study of Segment Anything Model on Zero-Shot Medical Segmentation](https://mlsanity.com/paper/2304.12637/) (2023)
- [VisionFM: a Multi-Modal Multi-Task Vision Foundation Model for Generalist Ophthalmic Artificial Intelligence](https://mlsanity.com/paper/2310.04992/) (2023)
- [Learning Two-Stream CNN for Multi-Modal Age-related Macular Degeneration Categorization](https://mlsanity.com/paper/2012.01879/) (2020)
- [TFormer: A throughout fusion transformer for multi-modal skin lesion diagnosis](https://mlsanity.com/paper/2211.11393/) (2022)
- [Automatic segmentation of the Foveal Avascular Zone in ophthalmological OCT-A images](https://mlsanity.com/paper/1811.10374/) (2018)
- [CauDR: A Causality-inspired Domain Generalization Framework for Fundus-based Diabetic Retinopathy Grading](https://mlsanity.com/paper/2309.15493/) (2023)
- [Deep-OCTA: Ensemble Deep Learning Approaches for Diabetic Retinopathy Analysis on OCTA Images](https://mlsanity.com/paper/2210.00515/) (2022)

## Abstract (arXiv)

Ultra-wide optical coherence tomography angiography (UW-OCTA) is an emerging imaging technique that offers significant advantages over traditional OCTA by providing an exceptionally wide scanning range of up to 24 x 20 mm2, covering both the anterior and posterior regions of the retina. However, the currently accessible UW-OCTA datasets suffer from limited comprehensive hierarchical information and corresponding disease annotations. To address this limitation, we have curated the pioneering M3OCTA dataset, which is the first multimodal (i.e., multilayer), multi-disease, and widest field-of-view UW-OCTA dataset. Furthermore, the effective utilization of multi-layer ultra-wide ocular vasculature information from UW-OCTA remains underdeveloped. To tackle this challenge, we propose the first cross-modal fusion framework that leverages multi-modal information for diagnosing multiple diseases. Through extensive experiments conducted on our openly available M3OCTA dataset, we demonstrate the effectiveness and superior performance of our method, both in fixed and varying modalities settings. The construction of the M3OCTA dataset, the first multimodal OCTA dataset encompassing multiple diseases, aims to advance research in the ophthalmic image analysis community.