# Cross-Modality Image Registration using a Training-Time Privileged Third Modality

arXiv: https://arxiv.org/abs/2207.12901 · HTML: https://mlsanity.com/paper/2207.12901/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 52 references that are in the map.
- First posted on arXiv 26 Jul 2022.

## Builds on (most cited first, 18 of 18)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [VoxelMorph: A Learning Framework for Deformable Medical Image Registration](https://mlsanity.com/paper/1809.05231/) (2018)
- [An Unsupervised Learning Model for Deformable Medical Image Registration](https://mlsanity.com/paper/1802.02604/) (2018)
- [Unsupervised Learning for Fast Probabilistic Diffeomorphic Registration](https://mlsanity.com/paper/1805.04605/) (2018)
- [Weakly-Supervised Convolutional Neural Networks for Multimodal Image Registration](https://mlsanity.com/paper/1807.03361/) (2018)
- [Unsupervised Deformable Registration for Multi-Modal Images via Disentangled Representations](https://mlsanity.com/paper/1903.09331/) (2019)
- [Joint Learning of Motion Estimation and Segmentation for Cardiac MR Image Sequences](https://mlsanity.com/paper/1806.04066/) (2018)
- [Image-and-Spatial Transformer Networks for Structure-Guided Image Registration](https://mlsanity.com/paper/1907.09200/) (2019)
- [Unsupervised Probabilistic Deformation Modeling for Robust Diffeomorphic Registration](https://mlsanity.com/paper/1804.07172/) (2018)
- [Person Re-identification with Metric Learning using Privileged Information](https://mlsanity.com/paper/1904.05005/) (2019)
- [A Deep Metric for Multimodal Registration](https://mlsanity.com/paper/1609.05396/) (2016)
- [DeepReg: a deep learning toolkit for medical image registration](https://mlsanity.com/paper/2011.02580/) (2020)
- [Automatic brain tumor grading from MRI data using convolutional neural networks and quality assessment](https://mlsanity.com/paper/1809.09468/) (2018)
- [Linear and Deformable Image Registration with 3D Convolutional Neural Networks](https://mlsanity.com/paper/1809.06226/) (2018)
- [Robust Multimodal Image Registration Using Deep Recurrent Reinforcement Learning](https://mlsanity.com/paper/2002.03733/) (2020)
- [Enhancing Label-Driven Deep Deformable Image Registration with Local Distance Metrics for State-of-the-Art Cardiac Motion Tracking](https://mlsanity.com/paper/1812.01859/) (2018)
- [A Comparative Analysis of Registration Tools: Traditional vs Deep Learning Approach on High Resolution Tissue Cleared Data](https://mlsanity.com/paper/1810.08315/) (2018)
- [Longitudinal Image Registration with Temporal-order and Subject-specificity Discrimination](https://mlsanity.com/paper/2008.13002/) (2020)

## Built on by (most cited first, 9 of 9)

- [ModeTv2: GPU-accelerated Motion Decomposition Transformer for Pairwise Optimization in Medical Image Registration](https://mlsanity.com/paper/2403.16526/) (2024)
- [T2-Only Prostate Cancer Prediction by Meta-Learning from Bi-Parametric MR Imaging](https://mlsanity.com/paper/2411.07416/) (2024)
- [Modality-Aware Feature Matching in Visual and Vision-Language Applications: A Comprehensive Survey](https://mlsanity.com/paper/2507.22791/) (2025)
- [Spatial Correspondence between Graph Neural Network-Segmented Images](https://mlsanity.com/paper/2303.06550/) (2023)
- [Deep learning in medical image registration: introduction and survey](https://mlsanity.com/paper/2309.00727/) (2023)
- [On the Degrees of Freedom of Gridded Control Points in Learning-Based Medical Image Registration](https://mlsanity.com/paper/2603.16940/) (2026)
- [Maximizing T2-Only Prostate Cancer Localization from Expected Diffusion Weighted Imaging](https://mlsanity.com/paper/2604.00985/) (2026)
- [CT-Guided Spatially-varying Regularization for Voxel-Wise Deformable Whole-Body PET Registration](https://mlsanity.com/paper/2604.22905/) (2026)
- [TRACE: Training-time Report-guided and Clinically Ordered Concept Editing](https://mlsanity.com/paper/2608.20809/) (2026)

## Abstract (arXiv)

In this work, we consider the task of pairwise cross-modality image registration, which may benefit from exploiting additional images available only at training time from an additional modality that is different to those being registered. As an example, we focus on aligning intra-subject multiparametric Magnetic Resonance (mpMR) images, between T2-weighted (T2w) scans and diffusion-weighted scans with high b-value (DWIhigh-b). For the application of localising tumours in mpMR images, diffusion scans with zero b-value (DWIb=0) are considered easier to register to T2w due to the availability of corresponding features. We propose a learning from privileged modality algorithm, using a training-only imaging modality DWIb=0, to support the challenging multi-modality registration problems. We present experimental results based on 369 sets of 3D multiparametric MRI images from 356 prostate cancer patients and report, with statistical significance, a lowered median target registration error of 4.34 mm, when registering the holdout DWIhigh-b and T2w image pairs, compared with that of 7.96 mm before registration. Results also show that the proposed learning-based registration networks enabled efficient registration with comparable or better accuracy, compared with a classical iterative algorithm and other tested learning-based methods with/without the additional modality. These compared algorithms also failed to produce any significantly improved alignment between DWIhigh-b and T2w in this challenging application.