# X-RAFT: Cross-Modal Non-Rigid Registration of Blue and White Light Neurosurgical Hyperspectral Images

arXiv: https://arxiv.org/abs/2507.07747 · HTML: https://mlsanity.com/paper/2507.07747/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 8 of its 15 references that are in the map.
- First posted on arXiv 10 Jul 2025.

## Builds on (most cited first, 8 of 8)

- [RAFT: Recurrent All-Pairs Field Transforms for Optical Flow](https://mlsanity.com/paper/2003.12039/) (2020)
- [A survey on deep learning in medical image registration: new technologies, uncertainty, evaluation metrics, and beyond](https://mlsanity.com/paper/2307.15615/) (2023)
- [Transferring Relative Monocular Depth to Surgical Vision with Temporal Consistency](https://mlsanity.com/paper/2403.06683/) (2024)
- [Breaking Modality Disparity: Harmonized Representation for Infrared and Visible Image Registration](https://mlsanity.com/paper/2304.05646/) (2023)
- [A self-supervised and adversarial approach to hyperspectral demosaicking and RGB reconstruction in surgical imaging](https://mlsanity.com/paper/2407.19282/) (2024)
- [Neural Fields for 3D Tracking of Anatomy and Surgical Instruments in Monocular Laparoscopic Video Clips](https://mlsanity.com/paper/2403.19265/) (2024)
- [Towards RGB-NIR Cross-modality Image Registration and Beyond](https://mlsanity.com/paper/2405.19914/) (2024)
- [Motion-Boundary-Driven Unsupervised Surgical Instrument Segmentation in Low-Quality Optical Flow](https://mlsanity.com/paper/2403.10039/) (2024)

## Abstract (arXiv)

Integration of hyperspectral imaging into fluorescence-guided neurosurgery has the potential to improve surgical decision making by providing quantitative fluorescence measurements in real-time. Quantitative fluorescence requires paired spectral data in fluorescence (blue light) and reflectance (white light) mode. Blue and white image acquisition needs to be performed sequentially in a potentially dynamic surgical environment. A key component to the fluorescence quantification process is therefore the ability to find dense cross-modal image correspondences between two hyperspectral images taken under these drastically different lighting conditions. We address this challenge with the introduction of X-RAFT, a Recurrent All-Pairs Field Transforms (RAFT) optical flow model modified for cross-modal inputs. We propose using distinct image encoders for each modality pair, and fine-tune these in a self-supervised manner using flow-cycle-consistency on our neurosurgical hyperspectral data. We show an error reduction of 36.6% across our evaluation metrics when comparing to a naive baseline and 27.83% reduction compared to an existing cross-modal optical flow method (CrossRAFT). Our code and models will be made publicly available after the review process.