# MeLo: Low-rank Adaptation is Better than Fine-tuning for Medical Image Diagnosis

arXiv: https://arxiv.org/abs/2311.08236 · HTML: https://mlsanity.com/paper/2311.08236/

## Key facts

- Cited by 33 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 14 references that are in the map.
- First posted on arXiv 14 Nov 2023.

## Builds on (most cited first, 8 of 8)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Reproducible scaling laws for contrastive language-image learning](https://mlsanity.com/paper/2212.07143/) (2022)
- [ChestX-ray8: Hospital-scale Chest X-ray Database and Benchmarks on Weakly-Supervised Classification and Localization of Common Thorax Diseases](https://mlsanity.com/paper/1705.02315/) (2017)
- [Scaling Vision Transformers to 22 Billion Parameters](https://mlsanity.com/paper/2302.05442/) (2023)
- [Scaling Vision Transformers to Gigapixel Images via Hierarchical Self-Supervised Learning](https://mlsanity.com/paper/2206.02647/) (2022)
- [Deep Learning for Medical Image Processing: Overview, Challenges and Future](https://mlsanity.com/paper/1704.06825/) (2017)

## Built on by (most cited first, 30 of 33)

- [Low-Rank Adaptation for Foundation Models: A Comprehensive Review](https://mlsanity.com/paper/2501.00365/) (2024)
- [Online-LoRA: Task-free Online Continual Learning via Low Rank Adaptation](https://mlsanity.com/paper/2411.05663/) (2024)
- [PanDA: Towards Panoramic Depth Anything with Unlabeled Panoramas and Mobius Spatial Augmentation](https://mlsanity.com/paper/2406.13378/) (2024)
- [Less Could Be Better: Parameter-efficient Fine-tuning Advances Medical Vision Foundation Models](https://mlsanity.com/paper/2401.12215/) (2024)
- [Sample-specific Masks for Visual Reprogramming-based Prompting](https://mlsanity.com/paper/2406.03150/) (2024)
- [Distillation of Diffusion Features for Semantic Correspondence](https://mlsanity.com/paper/2412.03512/) (2024)
- [Can Visual Foundation Models Achieve Long-term Point Tracking?](https://mlsanity.com/paper/2408.13575/) (2024)
- [Bayesian-guided Label Mapping for Visual Reprogramming](https://mlsanity.com/paper/2410.24018/) (2024)
- [Attention, Please! Revisiting Attentive Probing Through the Lens of Efficiency](https://mlsanity.com/paper/2506.10178/) (2025)
- [Parameter-Efficient Adaptation of Geospatial Foundation Models through Embedding Deflection](https://mlsanity.com/paper/2503.09493/) (2025)
- [Foundation Models in Medical Imaging: A Review and Outlook](https://mlsanity.com/paper/2506.09095/) (2025)
- [LoRTA: Low Rank Tensor Adaptation of Large Language Models](https://mlsanity.com/paper/2410.04060/) (2024)
- [Robust Bird's Eye View Segmentation by Adapting DINOv2](https://mlsanity.com/paper/2409.10228/) (2024)
- [Reprogramming Distillation for Medical Foundation Models](https://mlsanity.com/paper/2407.06504/) (2024)
- [Efficient Medical Vision-Language Alignment Through Adapting Masked Vision Models](https://mlsanity.com/paper/2506.08990/) (2025)
- [Efficient In-Context Medical Segmentation with Meta-driven Visual Prompt Selection](https://mlsanity.com/paper/2407.11188/) (2024)
- [SAM & SAM 2 in 3D Slicer: SegmentWithSAM Extension for Annotating Medical Images](https://mlsanity.com/paper/2408.15224/) (2024)
- [Two-stage Cytopathological Image Synthesis for Augmenting Cervical Abnormality Screening](https://mlsanity.com/paper/2402.14707/) (2024)
- [Certifying Adapters: Enabling and Enhancing the Certification of Classifier Adversarial Robustness](https://mlsanity.com/paper/2405.16036/) (2024)
- [ZALM3: Zero-Shot Enhancement of Vision-Language Alignment via In-Context Information in Multi-Turn Multimodal Medical Dialogue](https://mlsanity.com/paper/2409.17610/) (2024)
- [RECAST: Reparameterized, Compact weight Adaptation for Sequential Tasks](https://mlsanity.com/paper/2411.16870/) (2024)
- [Federated Learning for Large Models in Medical Imaging: A Comprehensive Review](https://mlsanity.com/paper/2508.20414/) (2025)
- [LQ-Adapter: ViT-Adapter with Learnable Queries for Gallbladder Cancer Detection from Ultrasound Image](https://mlsanity.com/paper/2412.00374/) (2024)
- [Data-Efficient Fine-Tuning of Vision-Language Models for Diagnosis of Alzheimer's Disease](https://mlsanity.com/paper/2509.07613/) (2025)
- [OU-CoViT: Copula-Enhanced Bi-Channel Multi-Task Vision Transformers with Dual Adaptation for OU-UWF Images](https://mlsanity.com/paper/2408.09395/) (2024)
- [Med-LEGO: Editing and Adapting toward Generalist Medical Image Diagnosis](https://mlsanity.com/paper/2503.01164/) (2025)
- [Efficient Self-Supervised Adaptation for Medical Image Analysis](https://mlsanity.com/paper/2503.18873/) (2025)
- [Enhancing Pre-Trained Model-Based Class-Incremental Learning through Neural Collapse](https://mlsanity.com/paper/2504.18437/) (2025)
- [BALR-SAM: Boundary-Aware Low-Rank Adaptation of SAM for Resource-Efficient Medical Image Segmentation](https://mlsanity.com/paper/2509.24204/) (2025)
- [ProGiDiff: Prompt-Guided Diffusion-Based Medical Image Segmentation](https://mlsanity.com/paper/2601.16060/) (2026)

## Abstract (arXiv)

The common practice in developing computer-aided diagnosis (CAD) models based on transformer architectures usually involves fine-tuning from ImageNet pre-trained weights. However, with recent advances in large-scale pre-training and the practice of scaling laws, Vision Transformers (ViT) have become much larger and less accessible to medical imaging communities. Additionally, in real-world scenarios, the deployments of multiple CAD models can be troublesome due to problems such as limited storage space and time-consuming model switching. To address these challenges, we propose a new method MeLo (Medical image Low-rank adaptation), which enables the development of a single CAD model for multiple clinical tasks in a lightweight manner. It adopts low-rank adaptation instead of resource-demanding fine-tuning. By fixing the weight of ViT models and only adding small low-rank plug-ins, we achieve competitive results on various diagnosis tasks across different imaging modalities using only a few trainable parameters. Specifically, our proposed method achieves comparable performance to fully fine-tuned ViT models on four distinct medical imaging datasets using about 0.17% trainable parameters. Moreover, MeLo adds only about 0.5MB of storage space and allows for extremely fast model switching in deployment and inference. Our source code and pre-trained weights are available on our website (https://absterzhu.github.io/melo.github.io/).