# Real-time multimodal image registration with partial intraoperative point-set data

arXiv: https://arxiv.org/abs/2109.05023 · HTML: https://mlsanity.com/paper/2109.05023/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 72 references that are in the map.
- First posted on arXiv 10 Sep 2021.

## Builds on (most cited first, 17 of 17)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation](https://mlsanity.com/paper/1612.00593/) (2016)
- [A Point Set Generation Network for 3D Object Reconstruction from a Single Image](https://mlsanity.com/paper/1612.00603/) (2016)
- [TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems](https://mlsanity.com/paper/1603.04467/) (2016)
- [Deep Closest Point: Learning Representations for Point Cloud Registration](https://mlsanity.com/paper/1905.03304/) (2019)
- [FlowNet3D: Learning Scene Flow in 3D Point Clouds](https://mlsanity.com/paper/1806.01411/) (2018)
- [PointNetLK: Robust & Efficient Point Cloud Registration using PointNet](https://mlsanity.com/paper/1903.05711/) (2019)
- [PRNet: Self-Supervised Learning for Partial-to-Partial Registration](https://mlsanity.com/paper/1910.12240/) (2019)
- [Weakly-Supervised Convolutional Neural Networks for Multimodal Image Registration](https://mlsanity.com/paper/1807.03361/) (2018)
- [Adversarial Uni- and Multi-modal Stream Networks for Multimodal Image Registration](https://mlsanity.com/paper/2007.02790/) (2020)
- [Learning Deep Similarity Metric for 3D MR-TRUS Registration](https://mlsanity.com/paper/1806.04548/) (2018)
- [Adversarial Image Registration with Application for MR and TRUS Image Fusion](https://mlsanity.com/paper/1804.11024/) (2018)
- [Multimodality Biomedical Image Registration using Free Point Transformer Networks](https://mlsanity.com/paper/2008.01885/) (2020)
- [Learning Deformable Point Set Registration with Regularized Dynamic Graph CNNs for Large Lung Motion in COPD Patients](https://mlsanity.com/paper/1909.07818/) (2019)
- [Robust Multimodal Image Registration Using Deep Recurrent Reinforcement Learning](https://mlsanity.com/paper/2002.03733/) (2020)
- [Prostate motion modelling using biomechanically-trained deep neural networks on unstructured nodes](https://mlsanity.com/paper/2007.04972/) (2020)

## Built on by (most cited first, 10 of 10)

- [Non-rigid Medical Image Registration using Physics-informed Neural Networks](https://mlsanity.com/paper/2302.10343/) (2023)
- [Deep Implicit Optimization enables Robust Learnable Features for Deformable Image Registration](https://mlsanity.com/paper/2406.07361/) (2024)
- [Meta-Learning Initializations for Interactive Medical Image Registration](https://mlsanity.com/paper/2210.15371/) (2022)
- [Coupled Laplacian Eigenmaps for Locally-Aware 3D Rigid Point Cloud Matching](https://mlsanity.com/paper/2402.17372/) (2024)
- [A Narrative Review of Image Processing Techniques Related to Prostate Ultrasound](https://mlsanity.com/paper/2407.00678/) (2024)
- [GERA: Geometric Embedding for Efficient Point Registration Analysis](https://mlsanity.com/paper/2410.00589/) (2024)
- [Learning Generalized Non-Rigid Multimodal Biomedical Image Registration from Generic Point Set Data](https://mlsanity.com/paper/2207.10994/) (2022)
- [UniRiT: Towards Few-Shot Non-Rigid Point Cloud Registration](https://mlsanity.com/paper/2410.22909/) (2024)
- [Salient Region Matching for Fully Automated MR-TRUS Registration](https://mlsanity.com/paper/2501.03510/) (2025)
- [On the Degrees of Freedom of Gridded Control Points in Learning-Based Medical Image Registration](https://mlsanity.com/paper/2603.16940/) (2026)

## Abstract (arXiv)

We present Free Point Transformer (FPT) - a deep neural network architecture for non-rigid point-set registration. Consisting of two modules, a global feature extraction module and a point transformation module, FPT does not assume explicit constraints based on point vicinity, thereby overcoming a common requirement of previous learning-based point-set registration methods. FPT is designed to accept unordered and unstructured point-sets with a variable number of points and uses a "model-free" approach without heuristic constraints. Training FPT is flexible and involves minimizing an intuitive unsupervised loss function, but supervised, semi-supervised, and partially- or weakly-supervised training are also supported. This flexibility makes FPT amenable to multimodal image registration problems where the ground-truth deformations are difficult or impossible to measure. In this paper, we demonstrate the application of FPT to non-rigid registration of prostate magnetic resonance (MR) imaging and sparsely-sampled transrectal ultrasound (TRUS) images. The registration errors were 4.71 mm and 4.81 mm for complete TRUS imaging and sparsely-sampled TRUS imaging, respectively. The results indicate superior accuracy to the alternative rigid and non-rigid registration algorithms tested and substantially lower computation time. The rapid inference possible with FPT makes it particularly suitable for applications where real-time registration is beneficial.