LLaMA-Reg: Using LLaMA 2 for Unsupervised Medical Image Registration
Not on the map. Why?
Abstract (arXiv)
Medical image registration is an essential topic in medical image analysis. In this paper, we propose a method for medical image registration using a pretrained large language model. We find that using the pretrained large language model to encode deep features of the medical images in the registration model can effectively improve image registration accuracy, indicating the great potential of the large language model in medical image registration tasks. We use dual encoders to perform deep feature extraction on image pairs and then input the features into the pretrained large language model. To adapt the large language model to our registration task, the weights of the large language model are frozen in the registration model, and an adapter is utilized to fine-tune the large language model, which aims at (a) mapping the visual tokens to the language space before the large language model computing, (b) project the modeled language tokens output from the large language model to the visual space. Our method combines output features from the fine-tuned large language model with the features output from each encoder layer to gradually generate the deformation fields required for registration in the decoder. To demonstrate the effectiveness of the large prediction model in registration tasks, we conducted experiments on knee and brain MRI and achieved state-of-the-art results.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 23 of its 43 references that are in the map. First posted on arXiv 29 May 2024.
Builds on
23 of 43 references.
References · 23
- 2020Language Models are Few-Shot Learnerscited by 32,199
- 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
- 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
- 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
- 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
- 2021Learning to Prompt for Vision-Language Modelscited by 2,269
- 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
- 2015Spatial Transformer Networkscited by 1,423
- 2018VoxelMorph: A Learning Framework for Deformable Medical Image Registrationcited by 657
- 2019UNet++: Redesigning Skip Connections to Exploit Multiscale Features in Image Segmentationcited by 583
- 2017Learning multiple visual domains with residual adapterscited by 553
- 2019Unsupervised Learning of Probabilistic Diffeomorphic Registration for Images and Surfacescited by 186
11 more
- 2020Large Deformation Diffeomorphic Image Registration with Laplacian Pyramid Networkscited by 136
- 2021Learn2Reg: comprehensive multi-task medical image registration challenge, dataset and evaluation in the era of deep learningcited by 125
- 2020Fast Symmetric Diffeomorphic Image Registration with Convolutional Neural Networkscited by 107
- 2019Recursive Cascaded Networks for Unsupervised Medical Image Registrationcited by 105
- 2021ViT-V-Net: Vision Transformer for Unsupervised Volumetric Medical Image Registrationcited by 87
- 2019Unsupervised 3D End-to-End Medical Image Registration with Volume Tweening Networkcited by 62
- 2019Dual-Stream Pyramid Registration Networkcited by 52
- 2023Linear Spaces of Meanings: Compositional Structures in Vision-Language Modelscited by 48
- 2022XMorpher: Full Transformer for Deformable Medical Image Registration via Cross Attentioncited by 46
- 2022Non-iterative Coarse-to-fine Registration based on Single-pass Deep Cumulative Learningcited by 20
- 2022Symmetric Transformer-based Network for Unsupervised Image Registrationcited by 9
Built on by
Citing papers per year: 2025: 2.
2 papers.