# Decoupling Clinical and Class-Agnostic Features for Reliable Few-Shot Adaptation under Shift

arXiv: https://arxiv.org/abs/2509.09397 · HTML: https://mlsanity.com/paper/2509.09397/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 13 of its 27 references that are in the map.
- First posted on arXiv 11 Sep 2025.

## Placed in

- Multimodal LLMs / vision-language models / [Medical vision-language models](https://mlsanity.com/mllm/medical-vision-language-models/)

## Builds on (most cited first, 13 of 13)

- [Conditional Prompt Learning for Vision-Language Models](https://mlsanity.com/paper/2203.05557/) (2022)
- [MaPLe: Multi-modal Prompt Learning](https://mlsanity.com/paper/2210.03117/) (2022)
- [Domain Generalization for Medical Image Analysis: A Review](https://mlsanity.com/paper/2310.08598/) (2023)
- [Learning Disentangled Representations in the Imaging Domain](https://mlsanity.com/paper/2108.12043/) (2021)
- [Domain Generalization on Medical Imaging Classification using Episodic Training with Task Augmentation](https://mlsanity.com/paper/2106.06908/) (2021)
- [Few-shot Adaptation of Medical Vision-Language Models](https://mlsanity.com/paper/2409.03868/) (2024)
- [When More is Less: Incorporating Additional Datasets Can Hurt Performance By Introducing Spurious Correlations](https://mlsanity.com/paper/2308.04431/) (2023)
- [Test-Time Low Rank Adaptation via Confidence Maximization for Zero-Shot Generalization of Vision-Language Models](https://mlsanity.com/paper/2407.15913/) (2024)
- [A comprehensive and easy-to-use multi-domain multi-task medical imaging meta-dataset](https://mlsanity.com/paper/2404.16000/) (2024)
- [On the Robustness of Medical Vision-Language Models: Are they Truly Generalizable?](https://mlsanity.com/paper/2505.15425/) (2025)
- [Noise is an Efficient Learner for Zero-Shot Vision-Language Models](https://mlsanity.com/paper/2502.06019/) (2025)
- [DiMPLe -- Disentangled Multi-Modal Prompt Learning: Enhancing Out-Of-Distribution Alignment with Invariant and Spurious Feature Separation](https://mlsanity.com/paper/2506.21237/) (2025)
- [Can language-guided unsupervised adaptation improve medical image classification using unpaired images and texts?](https://mlsanity.com/paper/2409.02729/) (2024)

## Built on by (most cited first, 3 of 3)

- [Lost in Volume: The CT-SpatialVQA Benchmark for Evaluating Semantic-Spatial Understanding of 3D Medical Vision-Language Models](https://mlsanity.com/paper/2605.08787/) (2026)
- [Stride-Net: Fairness-Aware Disentangled Representation Learning for Chest X-Ray Diagnosis](https://mlsanity.com/paper/2602.10875/) (2026)
- [Autonomous Drift Learning in Data Streams: A Unified Perspective](https://mlsanity.com/paper/2605.01295/) (2026)

## Abstract (arXiv)

Medical vision-language models (VLMs) offer promise for clinical decision support, yet their reliability under distribution shifts remains a major concern for safe deployment. These models often learn task-agnostic correlations due to variability in imaging protocols and free-text reports, limiting their generalizability and increasing the risk of failure in real-world settings. We propose DRiFt, a structured feature decoupling framework that explicitly separates clinically relevant signals from task-agnostic noise using parameter-efficient tuning (LoRA) and learnable prompt tokens. To enhance cross-modal alignment and reduce uncertainty, we curate high-quality, clinically grounded image-text pairs by generating captions for a diverse medical dataset. Our approach improves in-distribution performance by +11.4% Top-1 accuracy and +3.3% Macro-F1 over prior prompt-based methods, while maintaining strong robustness across unseen datasets. Ablation studies reveal that disentangling task-relevant features and careful alignment significantly enhance model generalization and reduce unpredictable behavior under domain shift. These insights contribute toward building safer, more trustworthy VLMs for clinical use. The code is available at https://github.com/rumaima/DRiFt.