# Text-guided Foundation Model Adaptation for Long-Tailed Medical Image Classification

arXiv: https://arxiv.org/abs/2408.14770 · HTML: https://mlsanity.com/paper/2408.14770/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 24 references that are in the map.
- First posted on arXiv 27 Aug 2024.

## Builds on (most cited first, 18 of 18)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Decoupling Representation and Classifier for Long-Tailed Recognition](https://mlsanity.com/paper/1910.09217/) (2019)
- [Skin Lesion Analysis Toward Melanoma Detection 2018: A Challenge Hosted by the International Skin Imaging Collaboration (ISIC)](https://mlsanity.com/paper/1902.03368/) (2019)
- [MedCLIP: Contrastive Learning from Unpaired Medical Images and Text](https://mlsanity.com/paper/2210.10163/) (2022)
- [Balanced Meta-Softmax for Long-Tailed Visual Recognition](https://mlsanity.com/paper/2007.10740/) (2020)
- [Decoupled Knowledge Distillation](https://mlsanity.com/paper/2203.08679/) (2022)
- [Long-tailed Recognition by Routing Diverse Distribution-Aware Experts](https://mlsanity.com/paper/2010.01809/) (2020)
- [Parametric Contrastive Learning](https://mlsanity.com/paper/2107.12028/) (2021)
- [Balanced Contrastive Learning for Long-Tailed Visual Recognition](https://mlsanity.com/paper/2207.09052/) (2022)
- [ResLT: Residual Learning for Long-tailed Recognition](https://mlsanity.com/paper/2101.10633/) (2021)
- [A Foundation Language-Image Model of the Retina (FLAIR): Encoding Expert Knowledge in Text Supervision](https://mlsanity.com/paper/2308.07898/) (2023)
- [Learning Fast Sample Re-weighting Without Reward Data](https://mlsanity.com/paper/2109.03216/) (2021)
- [A Simple Long-Tailed Recognition Baseline via Vision-Language Model](https://mlsanity.com/paper/2111.14745/) (2021)
- [Balanced-MixUp for Highly Imbalanced Medical Image Classification](https://mlsanity.com/paper/2109.09850/) (2021)
- [Generalized Parametric Contrastive Learning](https://mlsanity.com/paper/2209.12400/) (2022)
- [ProCo: Prototype-aware Contrastive Learning for Long-tailed Medical Image Classification](https://mlsanity.com/paper/2209.00183/) (2022)
- [Uni4Eye: Unified 2D and 3D Self-supervised Pre-training via Masked Image Modeling Transformer for Ophthalmic Image Classification](https://mlsanity.com/paper/2203.04614/) (2022)
- [SSiT: Saliency-guided Self-supervised Image Transformer for Diabetic Retinopathy Grading](https://mlsanity.com/paper/2210.10969/) (2022)

## Built on by (most cited first, 5 of 5)

- [Benchmarking Foundation Models and Parameter-Efficient Fine-Tuning for Prognosis Prediction in Medical Imaging](https://mlsanity.com/paper/2506.18434/) (2025)
- [MONICA: Benchmarking on Long-tailed Medical Image Classification](https://mlsanity.com/paper/2410.02010/) (2024)
- [Adaptation of Foundation Models for Medical Image Analysis: Strategies, Challenges, and Future Directions](https://mlsanity.com/paper/2511.01284/) (2025)
- [TAGS: 3D Tumor-Adaptive Guidance for SAM](https://mlsanity.com/paper/2505.17096/) (2025)
- [ViTaL: A Multimodality Dataset and Benchmark for Multi-pathological Ovarian Tumor Recognition](https://mlsanity.com/paper/2507.04383/) (2025)

## Abstract (arXiv)

In medical contexts, the imbalanced data distribution in long-tailed datasets, due to scarce labels for rare diseases, greatly impairs the diagnostic accuracy of deep learning models. Recent multimodal text-image supervised foundation models offer new solutions to data scarcity through effective representation learning. However, their limited medical-specific pretraining hinders their performance in medical image classification relative to natural images. To address this issue, we propose a novel Text-guided Foundation model Adaptation for Long-Tailed medical image classification (TFA-LT). We adopt a two-stage training strategy, integrating representations from the foundation model using just two linear adapters and a single ensembler for balanced outcomes. Experimental results on two long-tailed medical image datasets validate the simplicity, lightweight and efficiency of our approach: requiring only 6.1% GPU memory usage of the current best-performing algorithm, our method achieves an accuracy improvement of up to 27.1%, highlighting the substantial potential of foundation model adaptation in this area.