# Cross-modal Attention for MRI and Ultrasound Volume Registration

arXiv: https://arxiv.org/abs/2107.04548 · HTML: https://mlsanity.com/paper/2107.04548/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 20 references that are in the map.
- First posted on arXiv 9 Jul 2021.

## Builds on (most cited first, 11 of 11)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [Aggregated Residual Transformations for Deep Neural Networks](https://mlsanity.com/paper/1611.05431/) (2016)
- [Non-local Neural Networks](https://mlsanity.com/paper/1711.07971/) (2017)
- [VoxelMorph: A Learning Framework for Deformable Medical Image Registration](https://mlsanity.com/paper/1809.05231/) (2018)
- [Deep Learning in Medical Image Registration: A Survey](https://mlsanity.com/paper/1903.02026/) (2019)
- [Weakly-Supervised Convolutional Neural Networks for Multimodal Image Registration](https://mlsanity.com/paper/1807.03361/) (2018)
- [End-to-End Unsupervised Deformable Image Registration with a Convolutional Neural Network](https://mlsanity.com/paper/1704.06065/) (2017)
- [Learning Deep Similarity Metric for 3D MR-TRUS Registration](https://mlsanity.com/paper/1806.04548/) (2018)
- [Adversarial Image Registration with Application for MR and TRUS Image Fusion](https://mlsanity.com/paper/1804.11024/) (2018)
- [Learning Multi-Modal Volumetric Prostate Registration with Weak Inter-Subject Spatial Correspondence](https://mlsanity.com/paper/2102.04938/) (2021)

## Built on by (most cited first, 21 of 21)

- [ModeT: Learning Deformable Image Registration via Motion Decomposition Transformer](https://mlsanity.com/paper/2306.05688/) (2023)
- [General Purpose Image Encoder DINOv2 for Medical Image Registration](https://mlsanity.com/paper/2402.15687/) (2024)
- [Residual Aligner Network](https://mlsanity.com/paper/2203.04290/) (2022)
- [A Flexible 2.5D Medical Image Segmentation Approach with In-Slice and Cross-Slice Attention](https://mlsanity.com/paper/2405.00130/) (2024)
- [Speckle2Self: Self-Supervised Ultrasound Speckle Reduction Without Clean Data](https://mlsanity.com/paper/2507.06828/) (2025)
- [Deformer: Towards Displacement Field Learning for Unsupervised Medical Image Registration](https://mlsanity.com/paper/2207.03180/) (2022)
- [Recursive Deformable Image Registration Network with Mutual Attention](https://mlsanity.com/paper/2206.01863/) (2022)
- [Epicardium Prompt-guided Real-time Cardiac Ultrasound Frame-to-volume Registration](https://mlsanity.com/paper/2406.14534/) (2024)
- [AutoFuse: Automatic Fusion Networks for Deformable Medical Image Registration](https://mlsanity.com/paper/2309.05271/) (2023)
- [VMambaMorph: a Multi-Modality Deformable Image Registration Framework based on Visual State Space Model with Cross-Scan Module](https://mlsanity.com/paper/2404.05105/) (2024)
- [Embedded Feature Similarity Optimization with Specific Parameter Initialization for 2D/3D Medical Image Registration](https://mlsanity.com/paper/2305.06252/) (2023)
- [ModeTv2: GPU-accelerated Motion Decomposition Transformer for Pairwise Optimization in Medical Image Registration](https://mlsanity.com/paper/2403.16526/) (2024)
- [A Narrative Review of Image Processing Techniques Related to Prostate Ultrasound](https://mlsanity.com/paper/2407.00678/) (2024)
- [Dual-Branch Squeeze-Fusion-Excitation Module for Cross-Modality Registration of Cardiac SPECT and CT](https://mlsanity.com/paper/2206.05278/) (2022)
- [On the Localization of Ultrasound Image Slices within Point Distribution Models](https://mlsanity.com/paper/2309.00372/) (2023)
- [Collaborative Quantization Embeddings for Intra-Subject Prostate MR Image Registration](https://mlsanity.com/paper/2207.06189/) (2022)
- [RS-MOCO: A deep learning-based topology-preserving image registration method for cardiac T1 mapping](https://mlsanity.com/paper/2410.11651/) (2024)
- [CFCML: A Coarse-to-Fine Crossmodal Learning Framework For Disease Diagnosis Using Multimodal Images and Tabular Data](https://mlsanity.com/paper/2603.20016/) (2026)
- [Cross Attention-guided Dense Network for Images Fusion](https://mlsanity.com/paper/2109.11393/) (2021)
- [The use of deep learning in interventional radiotherapy (brachytherapy): a review with a focus on open source and open data](https://mlsanity.com/paper/2205.07516/) (2022)
- [Explainable unsupervised multi-modal image registration using deep networks](https://mlsanity.com/paper/2308.01994/) (2023)

## Abstract (arXiv)

Prostate cancer biopsy benefits from accurate fusion of transrectal ultrasound (TRUS) and magnetic resonance (MR) images. In the past few years, convolutional neural networks (CNNs) have been proved powerful in extracting image features crucial for image registration. However, challenging applications and recent advances in computer vision suggest that CNNs are quite limited in its ability to understand spatial correspondence between features, a task in which the self-attention mechanism excels. This paper aims to develop a self-attention mechanism specifically for cross-modal image registration. Our proposed cross-modal attention block effectively maps each of the features in one volume to all features in the corresponding volume. Our experimental results demonstrate that a CNN network designed with the cross-modal attention block embedded outperforms an advanced CNN network 10 times of its size. We also incorporated visualization techniques to improve the interpretability of our network. The source code of our work is available at https://github.com/DIAL-RPI/Attention-Reg .