# General Vision Encoder Features as Guidance in Medical Image Registration

arXiv: https://arxiv.org/abs/2407.13311 · HTML: https://mlsanity.com/paper/2407.13311/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 43 references that are in the map.
- First posted on arXiv 18 Jul 2024.

## Builds on (most cited first, 26 of 26)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [VoxelMorph: A Learning Framework for Deformable Medical Image Registration](https://mlsanity.com/paper/1809.05231/) (2018)
- [An Introduction to Vision-Language Modeling](https://mlsanity.com/paper/2405.17247/) (2024)
- [On the Challenges and Perspectives of Foundation Models for Medical Image Analysis](https://mlsanity.com/paper/2306.05705/) (2023)
- [Deep Learning in Medical Image Registration: A Survey](https://mlsanity.com/paper/1903.02026/) (2019)
- [Unsupervised Learning for Fast Probabilistic Diffeomorphic Registration](https://mlsanity.com/paper/1805.04605/) (2018)
- [Merlin: A Computed Tomography Vision-Language Foundation Model and Dataset](https://mlsanity.com/paper/2406.06512/) (2024)
- [Efficient Multimodal Large Language Models: A Survey](https://mlsanity.com/paper/2405.10739/) (2024)
- [Foundational Models in Medical Imaging: A Comprehensive Survey and Future Vision](https://mlsanity.com/paper/2310.18689/) (2023)
- [Learning Conditional Deformable Templates with Convolutional Networks](https://mlsanity.com/paper/1908.02738/) (2019)
- [uniGradICON: A Foundation Model for Medical Image Registration](https://mlsanity.com/paper/2403.05780/) (2024)
- [Surgical-DINO: Adapter Learning of Foundation Models for Depth Estimation in Endoscopic Surgery](https://mlsanity.com/paper/2401.06013/) (2024)
- [Are Natural Domain Foundation Models Useful for Medical Image Classification?](https://mlsanity.com/paper/2310.19522/) (2023)
- [Foundation Models for Video Understanding: A Survey](https://mlsanity.com/paper/2405.03770/) (2024)
- [CoMIR: Contrastive Multimodal Image Representation for Registration](https://mlsanity.com/paper/2006.06325/) (2020)
- [Towards Vision-Language Geo-Foundation Model: A Survey](https://mlsanity.com/paper/2406.09385/) (2024)
- [Learning Deep Similarity Metric for 3D MR-TRUS Registration](https://mlsanity.com/paper/1806.04548/) (2018)
- [The Evolution of Multimodal Model Architectures](https://mlsanity.com/paper/2405.17927/) (2024)
- [A Deep Metric for Multimodal Registration](https://mlsanity.com/paper/1609.05396/) (2016)
- [AirLab: Autograd Image Registration Laboratory](https://mlsanity.com/paper/1806.09907/) (2018)
- [General Purpose Image Encoder DINOv2 for Medical Image Registration](https://mlsanity.com/paper/2402.15687/) (2024)
- [MAD: Modality Agnostic Distance Measure for Image Registration](https://mlsanity.com/paper/2309.02875/) (2023)
- [DeepSim: Semantic similarity metrics for learned image registration](https://mlsanity.com/paper/2011.05735/) (2020)

## Built on by (most cited first, 1 of 1)

- [Guiding Registration with Emergent Similarity from Pre-Trained Diffusion Models](https://mlsanity.com/paper/2506.02419/) (2025)

## Abstract (arXiv)

General vision encoders like DINOv2 and SAM have recently transformed computer vision. Even though they are trained on natural images, such encoder models have excelled in medical imaging, e.g., in classification, segmentation, and registration. However, no in-depth comparison of different state-of-the-art general vision encoders for medical registration is available. In this work, we investigate how well general vision encoder features can be used in the dissimilarity metrics for medical image registration. We explore two encoders that were trained on natural images as well as one that was fine-tuned on medical data. We apply the features within the well-established B-spline FFD registration framework. In extensive experiments on cardiac cine MRI data, we find that using features as additional guidance for conventional metrics improves the registration quality. The code is available at github.com/compai-lab/2024-miccai-koegl.