# Multi-modal Retinal Image Registration Using a Keypoint-Based Vessel Structure Aligning Network

arXiv: https://arxiv.org/abs/2207.10506 · HTML: https://mlsanity.com/paper/2207.10506/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 26 references that are in the map.
- First posted on arXiv 21 Jul 2022.

## Builds on (most cited first, 15 of 15)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [SuperGlue: Learning Feature Matching with Graph Neural Networks](https://mlsanity.com/paper/1911.11763/) (2019)
- [SuperPoint: Self-Supervised Interest Point Detection and Description](https://mlsanity.com/paper/1712.07629/) (2017)
- [Kornia: an Open Source Differentiable Computer Vision Library for PyTorch](https://mlsanity.com/paper/1910.02190/) (2019)
- [Learning to Find Good Correspondences](https://mlsanity.com/paper/1711.05971/) (2017)
- [LF-Net: Learning Local Features from Images](https://mlsanity.com/paper/1805.09662/) (2018)
- [ALIKE: Accurate and Lightweight Keypoint Detection and Descriptor Extraction](https://mlsanity.com/paper/2112.02906/) (2021)
- [Numerical Coordinate Regression with Convolutional Neural Networks](https://mlsanity.com/paper/1801.07372/) (2018)
- [Unsupervised Multi-Modal Image Registration via Geometry Preserving Image-to-Image Translation](https://mlsanity.com/paper/2003.08073/) (2020)
- [Image-and-Spatial Transformer Networks for Structure-Guided Image Registration](https://mlsanity.com/paper/1907.09200/) (2019)
- [GLAMpoints: Greedily Learned Accurate Match points](https://mlsanity.com/paper/1908.06812/) (2019)
- [Deep Keypoint-Based Camera Pose Estimation with Geometric Constraints](https://mlsanity.com/paper/2007.15122/) (2020)
- [A Keypoint Detection and Description Network Based on the Vessel Structure for Multi-Modal Retinal Image Registration](https://mlsanity.com/paper/2201.02242/) (2022)

## Built on by (most cited first, 7 of 7)

- [MIFNet: Learning Modality-Invariant Features for Generalizable Multimodal Image Matching](https://mlsanity.com/paper/2501.11299/) (2025)
- [MEMO: Dataset and Methods for Robust Multimodal Retinal Image Registration with Large or Small Vessel Density Differences](https://mlsanity.com/paper/2309.14550/) (2023)
- [Retinal IPA: Iterative KeyPoints Alignment for Multimodal Retinal Imaging](https://mlsanity.com/paper/2407.18362/) (2024)
- [A Vessel-Segmentation-Based CycleGAN for Unpaired Multi-modal Retinal Image Synthesis](https://mlsanity.com/paper/2306.02901/) (2023)
- [Cross-modal Fundus Image Registration under Large FoV Disparity](https://mlsanity.com/paper/2512.12657/) (2025)
- [Coarse-to-Fine Non-rigid Multi-modal Image Registration for Historical Panel Paintings based on Crack Structures](https://mlsanity.com/paper/2601.16348/) (2026)
- [Quasi-multimodal-based pathophysiological feature learning for retinal disease diagnosis](https://mlsanity.com/paper/2602.03622/) (2026)

## Abstract (arXiv)

In ophthalmological imaging, multiple imaging systems, such as color fundus, infrared, fluorescein angiography, optical coherence tomography (OCT) or OCT angiography, are often involved to make a diagnosis of retinal disease. Multi-modal retinal registration techniques can assist ophthalmologists by providing a pixel-based comparison of aligned vessel structures in images from different modalities or acquisition times. To this end, we propose an end-to-end trainable deep learning method for multi-modal retinal image registration. Our method extracts convolutional features from the vessel structure for keypoint detection and description and uses a graph neural network for feature matching. The keypoint detection and description network and graph neural network are jointly trained in a self-supervised manner using synthetic multi-modal image pairs and are guided by synthetically sampled ground truth homographies. Our method demonstrates higher registration accuracy as competing methods for our synthetic retinal dataset and generalizes well for our real macula dataset and a public fundus dataset.