# Label-driven weakly-supervised learning for multimodal deformable image registration

arXiv: https://arxiv.org/abs/1711.01666 · HTML: https://mlsanity.com/paper/1711.01666/

## Key facts

- Cited by 34 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 5 Nov 2017.

## Built on by (most cited first, 30 of 34)

- [Recent advances and clinical applications of deep learning in medical image analysis](https://mlsanity.com/paper/2105.13381/) (2021)
- [A survey on deep learning in medical image registration: new technologies, uncertainty, evaluation metrics, and beyond](https://mlsanity.com/paper/2307.15615/) (2023)
- [Affine Medical Image Registration with Coarse-to-Fine Vision Transformer](https://mlsanity.com/paper/2203.15216/) (2022)
- [A Robust and Interpretable Deep Learning Framework for Multi-modal Registration via Keypoints](https://mlsanity.com/paper/2304.09941/) (2023)
- [The Impact of Machine Learning on 2D/3D Registration for Image-guided Interventions: A Systematic Review and Perspective](https://mlsanity.com/paper/2108.02238/) (2021)
- [Anatomy-aware and acquisition-agnostic joint registration with SynthMorph](https://mlsanity.com/paper/2301.11329/) (2023)
- [Prototypical few-shot segmentation for cross-institution male pelvic structures with spatial registration](https://mlsanity.com/paper/2209.05160/) (2022)
- [Enhancing Medical Image Registration via Appearance Adjustment Networks](https://mlsanity.com/paper/2103.05213/) (2021)
- [Light-weight Deformable Registration using Adversarial Learning with Distilling Knowledge](https://mlsanity.com/paper/2110.01293/) (2021)
- [Joint Registration and Segmentation via Multi-Task Learning for Adaptive Radiotherapy of Prostate Cancer](https://mlsanity.com/paper/2105.01844/) (2021)
- [Deep Learning for Medical Image Registration: A Comprehensive Review](https://mlsanity.com/paper/2204.11341/) (2022)
- [Unsupervised Echocardiography Registration through Patch-based MLPs and Transformers](https://mlsanity.com/paper/2211.11687/) (2022)
- [Learning Multi-Modal Volumetric Prostate Registration with Weak Inter-Subject Spatial Correspondence](https://mlsanity.com/paper/2102.04938/) (2021)
- [TRUSTED: The Paired 3D Transabdominal Ultrasound and CT Human Data for Kidney Segmentation and Registration Research](https://mlsanity.com/paper/2310.12646/) (2023)
- [A Narrative Review of Image Processing Techniques Related to Prostate Ultrasound](https://mlsanity.com/paper/2407.00678/) (2024)
- [CAR-Net: Unsupervised Co-Attention Guided Registration Network for Joint Registration and Structure Learning](https://mlsanity.com/paper/2106.06637/) (2021)
- [Semi-weakly-supervised neural network training for medical image registration](https://mlsanity.com/paper/2402.10728/) (2024)
- [SAMReg: SAM-enabled Image Registration with ROI-based Correspondence](https://mlsanity.com/paper/2410.14083/) (2024)
- [Atlas-Based Segmentation of Intracochlear Anatomy in Metal Artifact Affected CT Images of the Ear with Co-trained Deep Neural Networks](https://mlsanity.com/paper/2107.03987/) (2021)
- [Spatial Correspondence between Graph Neural Network-Segmented Images](https://mlsanity.com/paper/2303.06550/) (2023)
- [Retrospective Motion Correction in Gradient Echo MRI by Explicit Motion Estimation Using Deep CNNs](https://mlsanity.com/paper/2303.17239/) (2023)
- [Stationary Velocity Fields on Matrix Groups for Deformable Image Registration](https://mlsanity.com/paper/2410.10997/) (2024)
- [SynStitch: a Self-Supervised Learning Network for Ultrasound Image Stitching Using Synthetic Training Pairs and Indirect Supervision](https://mlsanity.com/paper/2411.06750/) (2024)
- [D-Net: Siamese based Network with Mutual Attention for Volume Alignment](https://mlsanity.com/paper/2101.10248/) (2021)
- [The use of deep learning in interventional radiotherapy (brachytherapy): a review with a focus on open source and open data](https://mlsanity.com/paper/2205.07516/) (2022)
- [Multi-scale, Data-driven and Anatomically Constrained Deep Learning Image Registration for Adult and Fetal Echocardiography](https://mlsanity.com/paper/2309.00831/) (2023)
- [PatchMorph: A Stochastic Deep Learning Approach for Unsupervised 3D Brain Image Registration with Small Patches](https://mlsanity.com/paper/2312.06958/) (2023)
- [Weakly supervised alignment and registration of MR-CT for cervical cancer radiotherapy](https://mlsanity.com/paper/2405.12850/) (2024)
- [Multimodal Deformable Image Registration for Long-COVID Analysis Based on Progressive Alignment and Multi-perspective Loss](https://mlsanity.com/paper/2406.15172/) (2024)
- [Salient Region Matching for Fully Automated MR-TRUS Registration](https://mlsanity.com/paper/2501.03510/) (2025)

## Abstract (arXiv)

Spatially aligning medical images from different modalities remains a challenging task, especially for intraoperative applications that require fast and robust algorithms. We propose a weakly-supervised, label-driven formulation for learning 3D voxel correspondence from higher-level label correspondence, thereby bypassing classical intensity-based image similarity measures. During training, a convolutional neural network is optimised by outputting a dense displacement field (DDF) that warps a set of available anatomical labels from the moving image to match their corresponding counterparts in the fixed image. These label pairs, including solid organs, ducts, vessels, point landmarks and other ad hoc structures, are only required at training time and can be spatially aligned by minimising a cross-entropy function of the warped moving label and the fixed label. During inference, the trained network takes a new image pair to predict an optimal DDF, resulting in a fully-automatic, label-free, real-time and deformable registration. For interventional applications where large global transformation prevails, we also propose a neural network architecture to jointly optimise the global- and local displacements. Experiment results are presented based on cross-validating registrations of 111 pairs of T2-weighted magnetic resonance images and 3D transrectal ultrasound images from prostate cancer patients with a total of over 4000 anatomical labels, yielding a median target registration error of 4.2 mm on landmark centroids and a median Dice of 0.88 on prostate glands.