# Are Vision Foundation Models Ready for Out-of-the-Box Medical Image Registration?

arXiv: https://arxiv.org/abs/2507.11569 · HTML: https://mlsanity.com/paper/2507.11569/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 31 references that are in the map.
- First posted on arXiv 15 Jul 2025.

## Builds on (most cited first, 13 of 13)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [VoxelMorph: A Learning Framework for Deformable Medical Image Registration](https://mlsanity.com/paper/1809.05231/) (2018)
- [Segment Anything Model for Medical Image Analysis: an Experimental Study](https://mlsanity.com/paper/2304.10517/) (2023)
- [CycleMorph: Cycle Consistent Unsupervised Deformable Image Registration](https://mlsanity.com/paper/2008.05772/) (2020)
- [Generalist Vision Foundation Models for Medical Imaging: A Case Study of Segment Anything Model on Zero-Shot Medical Segmentation](https://mlsanity.com/paper/2304.12637/) (2023)
- [MedCLIP-SAMv2: Towards Universal Text-Driven Medical Image Segmentation](https://mlsanity.com/paper/2409.19483/) (2024)
- [How to build the best medical image segmentation algorithm using foundation models: a comprehensive empirical study with Segment Anything Model](https://mlsanity.com/paper/2404.09957/) (2024)
- [Segment anything model 2: an application to 2D and 3D medical images](https://mlsanity.com/paper/2408.00756/) (2024)
- [MRI-CORE: A Foundation Model for Magnetic Resonance Imaging](https://mlsanity.com/paper/2506.12186/) (2025)
- [Breast density in MRI: an AI-based quantification and relationship to assessment in mammography](https://mlsanity.com/paper/2504.15192/) (2025)
- [GuidedMorph: Two-Stage Deformable Registration for Breast MRI](https://mlsanity.com/paper/2505.13414/) (2025)

## Built on by (most cited first, 6 of 6)

- [DINOv3 with Test-Time Training for Medical Image Registration](https://mlsanity.com/paper/2508.14809/) (2025)
- [VoxCor: Training-Free Volumetric Features for Multimodal Voxel Correspondence](https://mlsanity.com/paper/2605.13798/) (2026)
- [Lost in Distortion: Uncovering the Domain Gap Between Computer Vision and Brain Imaging -- A Study on Pretraining for Age Prediction](https://mlsanity.com/paper/2512.01310/) (2025)
- [GLIDE-Reg: Global-to-Local Deformable Registration Using Co-Optimized Foundation and Handcrafted Features](https://mlsanity.com/paper/2603.00218/) (2026)
- [Effective Feature Learning for 3D Medical Registration via Domain-Specialized DINO Pretraining](https://mlsanity.com/paper/2603.14086/) (2026)
- [HyperCT: Low-Rank Hypernet for Unified Chest CT Analysis](https://mlsanity.com/paper/2604.03224/) (2026)

## Abstract (arXiv)

Foundation models, pre-trained on large image datasets and capable of capturing rich feature representations, have recently shown potential for zero-shot image registration. However, their performance has mostly been tested in the context of rigid or less complex structures, such as the brain or abdominal organs, and it remains unclear whether these models can handle more challenging, deformable anatomy. Breast MRI registration is particularly difficult due to significant anatomical variation between patients, deformation caused by patient positioning, and the presence of thin and complex internal structure of fibroglandular tissue, where accurate alignment is crucial. Whether foundation model-based registration algorithms can address this level of complexity remains an open question. In this study, we provide a comprehensive evaluation of foundation model-based registration algorithms for breast MRI. We assess five pre-trained encoders, including DINO-v2, SAM, MedSAM, SSLSAM, and MedCLIP, across four key breast registration tasks that capture variations in different years and dates, sequences, modalities, and patient disease status (lesion versus no lesion). Our results show that foundation model-based algorithms such as SAM outperform traditional registration baselines for overall breast alignment, especially under large domain shifts, but struggle with capturing fine details of fibroglandular tissue. Interestingly, additional pre-training or fine-tuning on medical or breast-specific images in MedSAM and SSLSAM, does not improve registration performance and may even decrease it in some cases. Further work is needed to understand how domain-specific training influences registration and to explore targeted strategies that improve both global alignment and fine structure accuracy. We also publicly release our code at hrefhttps://github.com/mazurowski-lab/Foundation-based-regGithub.