# ViT-V-Net: Vision Transformer for Unsupervised Volumetric Medical Image Registration

arXiv: https://arxiv.org/abs/2104.06468 · HTML: https://mlsanity.com/paper/2104.06468/

## Key facts

- Cited by 87 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 22 references that are in the map.
- First posted on arXiv 13 Apr 2021.

## Builds on (most cited first, 14 of 14)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Non-local Neural Networks](https://mlsanity.com/paper/1711.07971/) (2017)
- [V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentation](https://mlsanity.com/paper/1606.04797/) (2016)
- [TransUNet: Transformers Make Strong Encoders for Medical Image Segmentation](https://mlsanity.com/paper/2102.04306/) (2021)
- [Spatial Transformer Networks](https://mlsanity.com/paper/1506.02025/) (2015)
- [Multi-Scale Context Aggregation by Dilated Convolutions](https://mlsanity.com/paper/1511.07122/) (2015)
- [Axial-DeepLab: Stand-Alone Axial-Attention for Panoptic Segmentation](https://mlsanity.com/paper/2003.07853/) (2020)
- [An Unsupervised Learning Model for Deformable Medical Image Registration](https://mlsanity.com/paper/1802.02604/) (2018)
- [End-to-End Unsupervised Deformable Image Registration with a Convolutional Neural Network](https://mlsanity.com/paper/1704.06065/) (2017)

## Built on by (most cited first, 30 of 87)

- [Transformers in Medical Imaging: A Survey](https://mlsanity.com/paper/2201.09873/) (2022)
- [Advances in Medical Image Analysis with Vision Transformers: A Comprehensive Review](https://mlsanity.com/paper/2301.03505/) (2023)
- [Is it Time to Replace CNNs with Transformers for Medical Images?](https://mlsanity.com/paper/2108.09038/) (2021)
- [A survey on deep learning in medical image registration: new technologies, uncertainty, evaluation metrics, and beyond](https://mlsanity.com/paper/2307.15615/) (2023)
- [Transformers in Medical Image Analysis: A Review](https://mlsanity.com/paper/2202.12165/) (2022)
- [One Model to Synthesize Them All: Multi-contrast Multi-scale Transformer for Missing Data Imputation](https://mlsanity.com/paper/2204.13738/) (2022)
- [U-Net vs Transformer: Is U-Net Outdated in Medical Image Registration?](https://mlsanity.com/paper/2208.04939/) (2022)
- [Affine Medical Image Registration with Coarse-to-Fine Vision Transformer](https://mlsanity.com/paper/2203.15216/) (2022)
- [Transforming medical imaging with Transformers? A comparative review of key properties, current progresses, and future perspectives](https://mlsanity.com/paper/2206.01136/) (2022)
- [Artificial General Intelligence for Medical Imaging Analysis](https://mlsanity.com/paper/2306.05480/) (2023)
- [ViTO: Vision Transformer-Operator](https://mlsanity.com/paper/2303.08891/) (2023)
- [Surface Vision Transformers: Attention-Based Modelling applied to Cortical Analysis](https://mlsanity.com/paper/2203.16414/) (2022)
- [ModeT: Learning Deformable Image Registration via Motion Decomposition Transformer](https://mlsanity.com/paper/2306.05688/) (2023)
- [A Recent Survey of Vision Transformers for Medical Image Segmentation](https://mlsanity.com/paper/2312.00634/) (2023)
- [Diffeomorphic Image Registration with Neural Velocity Field](https://mlsanity.com/paper/2202.12498/) (2022)
- [Coordinate Translator for Learning Deformable Medical Image Registration](https://mlsanity.com/paper/2203.03626/) (2022)
- [Medical Image Registration via Neural Fields](https://mlsanity.com/paper/2206.03111/) (2022)
- [Transformers in Healthcare: A Survey](https://mlsanity.com/paper/2307.00067/) (2023)
- [CyTran: A Cycle-Consistent Transformer with Multi-Level Consistency for Non-Contrast to Contrast CT Translation](https://mlsanity.com/paper/2110.06400/) (2021)
- [Toward Interpretable Sleep Stage Classification Using Cross-Modal Transformers](https://mlsanity.com/paper/2208.06991/) (2022)
- [Multi-scale Transformer Network with Edge-aware Pre-training for Cross-Modality MR Image Synthesis](https://mlsanity.com/paper/2212.01108/) (2022)
- [Pretrained ViTs Yield Versatile Representations For Medical Images](https://mlsanity.com/paper/2303.07034/) (2023)
- [A Novel Evaluation Framework for Image2Text Generation](https://mlsanity.com/paper/2408.01723/) (2024)
- [Vector Field Attention for Deformable Image Registration](https://mlsanity.com/paper/2407.10209/) (2024)
- [Image2Text2Image: A Novel Framework for Label-Free Evaluation of Image-to-Text Generation with Text-to-Image Diffusion Models](https://mlsanity.com/paper/2411.05706/) (2024)
- [Symmetric Transformer-based Network for Unsupervised Image Registration](https://mlsanity.com/paper/2204.13575/) (2022)
- [Deformable Cross-Attention Transformer for Medical Image Registration](https://mlsanity.com/paper/2303.06179/) (2023)
- [PC-SwinMorph: Patch Representation for Unsupervised Medical Image Registration and Segmentation](https://mlsanity.com/paper/2203.05684/) (2022)
- [Deformer: Towards Displacement Field Learning for Unsupervised Medical Image Registration](https://mlsanity.com/paper/2207.03180/) (2022)
- [3D Inception-Based TransMorph: Pre- and Post-operative Multi-contrast MRI Registration in Brain Tumors](https://mlsanity.com/paper/2212.04579/) (2022)

## Abstract (arXiv)

In the last decade, convolutional neural networks (ConvNets) have dominated and achieved state-of-the-art performances in a variety of medical imaging applications. However, the performances of ConvNets are still limited by lacking the understanding of long-range spatial relations in an image. The recently proposed Vision Transformer (ViT) for image classification uses a purely self-attention-based model that learns long-range spatial relations to focus on the relevant parts of an image. Nevertheless, ViT emphasizes the low-resolution features because of the consecutive downsamplings, result in a lack of detailed localization information, making it unsuitable for image registration. Recently, several ViT-based image segmentation methods have been combined with ConvNets to improve the recovery of detailed localization information. Inspired by them, we present ViT-V-Net, which bridges ViT and ConvNet to provide volumetric medical image registration. The experimental results presented here demonstrate that the proposed architecture achieves superior performance to several top-performing registration methods.