# MEMO: Dataset and Methods for Robust Multimodal Retinal Image Registration with Large or Small Vessel Density Differences

arXiv: https://arxiv.org/abs/2309.14550 · HTML: https://mlsanity.com/paper/2309.14550/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 80 references that are in the map.
- First posted on arXiv 25 Sep 2023.

## Builds on (most cited first, 16 of 16)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes](https://mlsanity.com/paper/1702.04405/) (2017)
- [Perceptual Losses for Real-Time Style Transfer and Super-Resolution](https://mlsanity.com/paper/1603.08155/) (2016)
- [SuperGlue: Learning Feature Matching with Graph Neural Networks](https://mlsanity.com/paper/1911.11763/) (2019)
- [SuperPoint: Self-Supervised Interest Point Detection and Description](https://mlsanity.com/paper/1712.07629/) (2017)
- [LoFTR: Detector-Free Local Feature Matching with Transformers](https://mlsanity.com/paper/2104.00680/) (2021)
- [From Coarse to Fine: Robust Hierarchical Localization at Large Scale](https://mlsanity.com/paper/1812.03506/) (2018)
- [HPatches: A benchmark and evaluation of handcrafted and learned local descriptors](https://mlsanity.com/paper/1704.05939/) (2017)
- [Learning Two-View Correspondences and Geometry Using Order-Aware Network](https://mlsanity.com/paper/1908.04964/) (2019)
- [Pixel-Adaptive Convolutional Neural Networks](https://mlsanity.com/paper/1904.05373/) (2019)
- [Deep Retinal Image Understanding](https://mlsanity.com/paper/1609.01103/) (2016)
- [Multi-modal Retinal Image Registration Using a Keypoint-Based Vessel Structure Aligning Network](https://mlsanity.com/paper/2207.10506/) (2022)
- [Multimodal Registration of Retinal Images Using Domain-Specific Landmarks and Vessel Enhancement](https://mlsanity.com/paper/1803.00951/) (2018)
- [A Keypoint Detection and Description Network Based on the Vessel Structure for Multi-Modal Retinal Image Registration](https://mlsanity.com/paper/2201.02242/) (2022)

## Built on by (most cited first, 3 of 3)

- [RetinaRegNet: A Zero-Shot Approach for Retinal Image Registration](https://mlsanity.com/paper/2404.16017/) (2024)
- [MIFNet: Learning Modality-Invariant Features for Generalizable Multimodal Image Matching](https://mlsanity.com/paper/2501.11299/) (2025)
- [Automated Erythrocyte Detection and Tracking for Retinal Blood Flow Quantification in Erythrocyte-Mediated Angiography](https://mlsanity.com/paper/2606.01006/) (2026)

## Abstract (arXiv)

The measurement of retinal blood flow (RBF) in capillaries can provide a powerful biomarker for the early diagnosis and treatment of ocular diseases. However, no single modality can determine capillary flowrates with high precision. Combining erythrocyte-mediated angiography (EMA) with optical coherence tomography angiography (OCTA) has the potential to achieve this goal, as EMA can measure the absolute 2D RBF of retinal microvasculature and OCTA can provide the 3D structural images of capillaries. However, multimodal retinal image registration between these two modalities remains largely unexplored. To fill this gap, we establish MEMO, the first public multimodal EMA and OCTA retinal image dataset. A unique challenge in multimodal retinal image registration between these modalities is the relatively large difference in vessel density (VD). To address this challenge, we propose a segmentation-based deep-learning framework (VDD-Reg) and a new evaluation metric (MSD), which provide robust results despite differences in vessel density. VDD-Reg consists of a vessel segmentation module and a registration module. To train the vessel segmentation module, we further designed a two-stage semi-supervised learning framework (LVD-Seg) combining supervised and unsupervised losses. We demonstrate that VDD-Reg outperforms baseline methods quantitatively and qualitatively for cases of both small VD differences (using the CF-FA dataset) and large VD differences (using our MEMO dataset). Moreover, VDD-Reg requires as few as three annotated vessel segmentation masks to maintain its accuracy, demonstrating its feasibility.