# Weakly-Supervised Convolutional Neural Networks for Multimodal Image Registration

arXiv: https://arxiv.org/abs/1807.03361 · HTML: https://mlsanity.com/paper/1807.03361/

## Key facts

- Cited by 90 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 Jul 2018.

## Built on by (most cited first, 30 of 90)

- [Recent advances and clinical applications of deep learning in medical image analysis](https://mlsanity.com/paper/2105.13381/) (2021)
- [LasHeR: A Large-scale High-diversity Benchmark for RGBT Tracking](https://mlsanity.com/paper/2104.13202/) (2021)
- [HyperMorph: Amortized Hyperparameter Learning for Image Registration](https://mlsanity.com/paper/2101.01035/) (2021)
- [DiffuseMorph: Unsupervised Deformable Image Registration Using Diffusion Model](https://mlsanity.com/paper/2112.05149/) (2021)
- [A survey on deep learning in medical image registration: new technologies, uncertainty, evaluation metrics, and beyond](https://mlsanity.com/paper/2307.15615/) (2023)
- [Learning the Effect of Registration Hyperparameters with HyperMorph](https://mlsanity.com/paper/2203.16680/) (2022)
- [A Robust and Interpretable Deep Learning Framework for Multi-modal Registration via Keypoints](https://mlsanity.com/paper/2304.09941/) (2023)
- [Joint Progressive and Coarse-to-fine Registration of Brain MRI via Deformation Field Integration and Non-Rigid Feature Fusion](https://mlsanity.com/paper/2109.12384/) (2021)
- [Cross-modal Attention for MRI and Ultrasound Volume Registration](https://mlsanity.com/paper/2107.04548/) (2021)
- [Prototypical few-shot segmentation for cross-institution male pelvic structures with spatial registration](https://mlsanity.com/paper/2209.05160/) (2022)
- [multiGradICON: A Foundation Model for Multimodal Medical Image Registration](https://mlsanity.com/paper/2408.00221/) (2024)
- [End-to-end Ultrasound Frame to Volume Registration](https://mlsanity.com/paper/2107.06449/) (2021)
- [Enhancing Medical Image Registration via Appearance Adjustment Networks](https://mlsanity.com/paper/2103.05213/) (2021)
- [Light-weight Deformable Registration using Adversarial Learning with Distilling Knowledge](https://mlsanity.com/paper/2110.01293/) (2021)
- [Unsupervised Multi-Modal Medical Image Registration via Discriminator-Free Image-to-Image Translation](https://mlsanity.com/paper/2204.13656/) (2022)
- [Real-time multimodal image registration with partial intraoperative point-set data](https://mlsanity.com/paper/2109.05023/) (2021)
- [Embedding Gradient-based Optimization in Image Registration Networks](https://mlsanity.com/paper/2112.03915/) (2021)
- [Residual Aligner Network](https://mlsanity.com/paper/2203.04290/) (2022)
- [X-Metric: An N-Dimensional Information-Theoretic Framework for Groupwise Registration and Deep Combined Computing](https://mlsanity.com/paper/2211.01631/) (2022)
- [Cross-Modality Image Registration using a Training-Time Privileged Third Modality](https://mlsanity.com/paper/2207.12901/) (2022)
- [Anatomy-guided Multimodal Registration by Learning Segmentation without Ground Truth: Application to Intraprocedural CBCT/MR Liver Segmentation and Registration](https://mlsanity.com/paper/2104.07056/) (2021)
- [Cross-Modality Multi-Atlas Segmentation via Deep Registration and Label Fusion](https://mlsanity.com/paper/2202.02000/) (2022)
- [Deep Learning for Medical Image Registration: A Comprehensive Review](https://mlsanity.com/paper/2204.11341/) (2022)
- [Non-rigid Medical Image Registration using Physics-informed Neural Networks](https://mlsanity.com/paper/2302.10343/) (2023)
- [Recursive Deformable Image Registration Network with Mutual Attention](https://mlsanity.com/paper/2206.01863/) (2022)
- [Aligning Multi-Sequence CMR Towards Fully Automated Myocardial Pathology Segmentation](https://mlsanity.com/paper/2302.03537/) (2023)
- [Unsupervised 3D registration through optimization-guided cyclical self-training](https://mlsanity.com/paper/2306.16997/) (2023)
- [Weakly-supervised Biomechanically-constrained CT/MRI Registration of the Spine](https://mlsanity.com/paper/2205.07568/) (2022)
- [AutoFuse: Automatic Fusion Networks for Deformable Medical Image Registration](https://mlsanity.com/paper/2309.05271/) (2023)
- [GRID: A Platform for General Robot Intelligence Development](https://mlsanity.com/paper/2310.00887/) (2023)

## Abstract (arXiv)

One of the fundamental challenges in supervised learning for multimodal image registration is the lack of ground-truth for voxel-level spatial correspondence. This work describes a method to infer voxel-level transformation from higher-level correspondence information contained in anatomical labels. We argue that such labels are more reliable and practical to obtain for reference sets of image pairs than voxel-level correspondence. Typical anatomical labels of interest may include solid organs, vessels, ducts, structure boundaries and other subject-specific ad hoc landmarks. The proposed end-to-end convolutional neural network approach aims to predict displacement fields to align multiple labelled corresponding structures for individual image pairs during the training, while only unlabelled image pairs are used as the network input for inference. We highlight the versatility of the proposed strategy, for training, utilising diverse types of anatomical labels, which need not to be identifiable over all training image pairs. At inference, the resulting 3D deformable image registration algorithm runs in real-time and is fully-automated without requiring any anatomical labels or initialisation. Several network architecture variants are compared for registering T2-weighted magnetic resonance images and 3D transrectal ultrasound images from prostate cancer patients. A median target registration error of 3.6 mm on landmark centroids and a median Dice of 0.87 on prostate glands are achieved from cross-validation experiments, in which 108 pairs of multimodal images from 76 patients were tested with high-quality anatomical labels.