# U-Net vs Transformer: Is U-Net Outdated in Medical Image Registration?

arXiv: https://arxiv.org/abs/2208.04939 · HTML: https://mlsanity.com/paper/2208.04939/

## Key facts

- Cited by 40 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 28 references that are in the map.
- First posted on arXiv 7 Aug 2022.

## Builds on (most cited first, 18 of 18)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentation](https://mlsanity.com/paper/1606.04797/) (2016)
- [VoxelMorph: A Learning Framework for Deformable Medical Image Registration](https://mlsanity.com/paper/1809.05231/) (2018)
- [RepVGG: Making VGG-style ConvNets Great Again](https://mlsanity.com/paper/2101.03697/) (2021)
- [Scaling Up Your Kernels to 31x31: Revisiting Large Kernel Design in CNNs](https://mlsanity.com/paper/2203.06717/) (2022)
- [An Unsupervised Learning Model for Deformable Medical Image Registration](https://mlsanity.com/paper/1802.02604/) (2018)
- [Unsupervised Learning for Fast Probabilistic Diffeomorphic Registration](https://mlsanity.com/paper/1805.04605/) (2018)
- [Fast Symmetric Diffeomorphic Image Registration with Convolutional Neural Networks](https://mlsanity.com/paper/2003.09514/) (2020)
- [Recursive Cascaded Networks for Unsupervised Medical Image Registration](https://mlsanity.com/paper/1907.12353/) (2019)
- [ViT-V-Net: Vision Transformer for Unsupervised Volumetric Medical Image Registration](https://mlsanity.com/paper/2104.06468/) (2021)
- [HyperMorph: Amortized Hyperparameter Learning for Image Registration](https://mlsanity.com/paper/2101.01035/) (2021)
- [CycleMorph: Cycle Consistent Unsupervised Deformable Image Registration](https://mlsanity.com/paper/2008.05772/) (2020)
- [Inverse-Consistent Deep Networks for Unsupervised Deformable Image Registration](https://mlsanity.com/paper/1809.03443/) (2018)
- [Learning a Model-Driven Variational Network for Deformable Image Registration](https://mlsanity.com/paper/2105.12227/) (2021)
- [Nesterov Accelerated ADMM for Fast Diffeomorphic Image Registration](https://mlsanity.com/paper/2109.12688/) (2021)

## Built on by (most cited first, 30 of 40)

- [A survey on deep learning in medical image registration: new technologies, uncertainty, evaluation metrics, and beyond](https://mlsanity.com/paper/2307.15615/) (2023)
- [Correlation-aware Coarse-to-fine MLPs for Deformable Medical Image Registration](https://mlsanity.com/paper/2406.00123/) (2024)
- [Fourier-Net: Fast Image Registration with Band-limited Deformation](https://mlsanity.com/paper/2211.16342/) (2022)
- [Deep Learning in Medical Image Registration: Magic or Mirage?](https://mlsanity.com/paper/2408.05839/) (2024)
- [Spatially Covariant Image Registration with Text Prompts](https://mlsanity.com/paper/2311.15607/) (2023)
- [Reverse Knowledge Distillation: Training a Large Model using a Small One for Retinal Image Matching on Limited Data](https://mlsanity.com/paper/2307.10698/) (2023)
- [MemWarp: Discontinuity-Preserving Cardiac Registration with Memorized Anatomical Filters](https://mlsanity.com/paper/2407.08093/) (2024)
- [Beyond the LUMIR challenge: The pathway to foundational registration models](https://mlsanity.com/paper/2505.24160/) (2025)
- [MambaMorph: a Mamba-based Framework for Medical MR-CT Deformable Registration](https://mlsanity.com/paper/2401.13934/) (2024)
- [Vector Field Attention for Deformable Image Registration](https://mlsanity.com/paper/2407.10209/) (2024)
- [Mamba? Catch The Hype Or Rethink What Really Helps for Image Registration](https://mlsanity.com/paper/2407.19274/) (2024)
- [AutoFuse: Automatic Fusion Networks for Deformable Medical Image Registration](https://mlsanity.com/paper/2309.05271/) (2023)
- [Full-resolution MLPs Empower Medical Dense Prediction](https://mlsanity.com/paper/2311.16707/) (2023)
- [Slicer Networks](https://mlsanity.com/paper/2401.09833/) (2024)
- [Large-kernel Attention for Efficient and Robust Brain Lesion Segmentation](https://mlsanity.com/paper/2308.07251/) (2023)
- [Decoder-Only Image Registration](https://mlsanity.com/paper/2402.03585/) (2024)
- [Deep Implicit Optimization enables Robust Learnable Features for Deformable Image Registration](https://mlsanity.com/paper/2406.07361/) (2024)
- [Learning deep abdominal CT registration through adaptive loss weighting and synthetic data generation](https://mlsanity.com/paper/2211.15717/) (2022)
- [Fourier-Net+: Leveraging Band-Limited Representation for Efficient 3D Medical Image Registration](https://mlsanity.com/paper/2307.02997/) (2023)
- [Learning with Constraint Learning: New Perspective, Solution Strategy and Various Applications](https://mlsanity.com/paper/2307.15257/) (2023)
- [Disentangling Progress in Medical Image Registration: Beyond Trend-Driven Architectures towards Domain-Specific Strategies](https://mlsanity.com/paper/2512.01913/) (2025)
- [Unsupervised Deformable Image Registration with Structural Nonparametric Smoothing](https://mlsanity.com/paper/2506.10813/) (2025)
- [Encoder-Only Image Registration](https://mlsanity.com/paper/2509.00451/) (2025)
- [DRIMET: Deep Registration for 3D Incompressible Motion Estimation in Tagged-MRI with Application to the Tongue](https://mlsanity.com/paper/2301.07234/) (2023)
- [MomentaMorph: Unsupervised Spatial-Temporal Registration with Momenta, Shooting, and Correction](https://mlsanity.com/paper/2308.02949/) (2023)
- [Toward Universal Medical Image Registration via Sharpness-Aware Meta-Continual Learning](https://mlsanity.com/paper/2406.17575/) (2024)
- [SACB-Net: Spatial-awareness Convolutions for Medical Image Registration](https://mlsanity.com/paper/2503.19592/) (2025)
- [Large Scale Unsupervised Brain MRI Image Registration Solution for Learn2Reg 2024](https://mlsanity.com/paper/2409.00917/) (2024)
- [A Scalable Distributed Framework for Multimodal GigaVoxel Image Registration](https://mlsanity.com/paper/2509.25044/) (2025)
- [Deep Spatiotemporal Clutter Filtering of Transthoracic Echocardiographic Images: Leveraging Contextual Attention and Residual Learning](https://mlsanity.com/paper/2401.13147/) (2024)

## Abstract (arXiv)

Due to their extreme long-range modeling capability, vision transformer-based networks have become increasingly popular in deformable image registration. We believe, however, that the receptive field of a 5-layer convolutional U-Net is sufficient to capture accurate deformations without needing long-range dependencies. The purpose of this study is therefore to investigate whether U-Net-based methods are outdated compared to modern transformer-based approaches when applied to medical image registration. For this, we propose a large kernel U-Net (LKU-Net) by embedding a parallel convolutional block to a vanilla U-Net in order to enhance the effective receptive field. On the public 3D IXI brain dataset for atlas-based registration, we show that the performance of the vanilla U-Net is already comparable with that of state-of-the-art transformer-based networks (such as TransMorph), and that the proposed LKU-Net outperforms TransMorph by using only 1.12% of its parameters and 10.8% of its mult-adds operations. We further evaluate LKU-Net on a MICCAI Learn2Reg 2021 challenge dataset for inter-subject registration, our LKU-Net also outperforms TransMorph on this dataset and ranks first on the public leaderboard as of the submission of this work. With only modest modifications to the vanilla U-Net, we show that U-Net can outperform transformer-based architectures on inter-subject and atlas-based 3D medical image registration. Code is available at https://github.com/xi-jia/LKU-Net.