mlsanityData to 7 Oct 2026

Mind the Discriminability Trap in Source-Free Cross-domain Few-shot Learning

arXiv 2603.13341 ↗First posted 7 Mar 2026cs.CV cs.AIcited by 4

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

Source-Free Cross-Domain Few-Shot Learning (SF-CDFSL) focuses on fine-tuning with limited training data from target domains (e.g., medical or satellite images), where Vision-Language Models (VLMs) such as CLIP and SigLIP have shown promising results. Current works in traditional visual models suggest that improving visual discriminability enhances performance. However, in VLM-based SF-CDFSL tasks, we find that strengthening visual-modal discriminability actually suppresses VLMs' performance. In this paper, we aim to delve into this phenomenon for an interpretation and a solution. By both theoretical and experimental proofs, our study reveals that fine-tuning with the typical cross-entropy loss (Lvlm) inherently includes a visual learning part and a cross-modal learning part, where the cross-modal part is crucial for rectifying the heavily disrupted modality misalignment in SF-CDFSL. However, we find that the visual learning essentially acts as a shortcut that encourages the model to reduce Lvlm without considering the cross-modal part, therefore hindering the cross-modal alignment and harming the performance. Based on this interpretation, we further propose an approach to address this problem: first, we perturb the visual learning to guide the model to focus on the cross-modal alignment. Then, we use the visual-text semantic relationships to gradually align the visual and textual modalities during the fine-tuning. Extensive experiments on various settings, backbones (CLIP, SigLip, PE-Core), and tasks (4 CDFSL datasets and 11 FSL datasets) show that we consistently set new state-of-the-art results. Code is available at https://github.com/zhenyuZ-HUST/CVPR26-Mind-the-Discriminability-Trap.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 66 of its 84 references that are in the map. First posted on arXiv 7 Mar 2026.

Builds on

66 of 84 references.

Closest ancestors · 17

Same direction, newest first

  1. 2025DynaPrompt: Dynamic Test-Time Prompt Tuning27 Jan
  2. 2024CLIP-CID: Efficient CLIP Distillation via Cluster-Instance Discrimination18 Aug
  3. 2024Low-Rank Few-Shot Adaptation of Vision-Language Models28 May
  4. 2024AMU-Tuning: Effective Logit Bias for CLIP-based Few-shot Learning13 Apr
  5. 2024PromptKD: Unsupervised Prompt Distillation for Vision-Language Models5 Mar
  6. 2023Prompt-based Distribution Alignment for Unsupervised Domain Adaptation15 Dec
  7. 2023TCP:Textual-based Class-aware Prompt tuning for Visual-Language Model30 Nov
  8. 2023Beyond Sole Strength: Customized Ensembles for Generalized Vision-Language Models28 Nov
  9. 2023Active Prompt Learning in Vision Language Models18 Nov
  10. 2023Visual-Language Prompt Tuning with Knowledge-guided Context Optimization23 Mar
  11. 2022Task Residual for Tuning Vision-Language Models18 Nov
  12. 2022Tip-Adapter: Training-free Adaption of CLIP for Few-shot Classification19 Jul
5 more
  1. 2022Prompt-aligned Gradient for Prompt Tuning30 May
  2. 2022Conditional Prompt Learning for Vision-Language Models10 Mar
  3. 2021Tip-Adapter: Training-free CLIP-Adapter for Better Vision-Language Modeling6 Nov
  4. 2021CLIP-Adapter: Better Vision-Language Models with Feature Adapters9 Oct
  5. 2021Learning to Prompt for Vision-Language Models2 Sep

Other references · 49

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  3. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  4. 2013Describing Textures in the Wildcited by 2,110
  5. 2020Shortcut Learning in Deep Neural Networkscited by 1,928
  6. 2018Unsupervised Feature Learning via Non-Parametric Instance-level Discriminationcited by 1,838
  7. 2013Fine-Grained Visual Classification of Aircraftcited by 1,701
  8. 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,488
  9. 2025SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Featurescited by 1,127
  10. 2017ChestX-ray8: Hospital-scale Chest X-ray Database and Benchmarks on Weakly-Supervised Classification and Localization of Common Thorax Diseasescited by 969
  11. 2022MaPLe: Multi-modal Prompt Learningcited by 685
  12. 2019A Closer Look at Few-shot Classificationcited by 679
37 more
  1. 2019Skin Lesion Analysis Toward Melanoma Detection 2018: A Challenge Hosted by the International Skin Imaging Collaboration (ISIC)cited by 657
  2. 2020Prototypical Contrastive Learning of Unsupervised Representationscited by 636
  3. 2022Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learningcited by 553
  4. 2023AdaLoRA: Adaptive Budget Allocation for Parameter-Efficient Fine-Tuningcited by 337
  5. 2025Perception Encoder: The best visual embeddings are not at the output of the networkcited by 315
  6. 2023Self-regulating Prompts: Foundational Model Adaptation without Forgettingcited by 272
  7. 2022DyLoRA: Parameter Efficient Tuning of Pre-trained Models using Dynamic Search-Free Low-Rank Adaptationcited by 229
  8. 2016Using Deep Learning for Image-Based Plant Disease Detectioncited by 216
  9. 2022Pushing the Limits of Simple Pipelines for Few-Shot Learning: External Data and Fine-Tuning Make a Differencecited by 123
  10. 2023One-for-All: Generalized LoRA for Parameter-Efficient Fine-tuningcited by 84
  11. 2023Delta-LoRA: Fine-Tuning High-Rank Parameters with the Delta of Low-Rank Matricescited by 72
  12. 2021Cross-Domain Few-Shot Classification via Adversarial Task Augmentationcited by 63
  13. 2024LP++: A Surprisingly Strong Linear Probe for Few-Shot CLIPcited by 60
  14. 2023StyleAdv: Meta Style Adversarial Training for Cross-Domain Few-Shot Learningcited by 53
  15. 2021Boosting the Generalization Capability in Cross-Domain Few-shot Learning via Noise-enhanced Supervised Autoencodercited by 51
  16. 2025Cross the Gap: Exposing the Intra-modal Misalignment in CLIP via Modality Inversioncited by 42
  17. 2023MetaDiff: Meta-Learning with Conditional Diffusion for Few-Shot Learningcited by 40
  18. 2022Adversarial Feature Augmentation for Cross-domain Few-shot Classificationcited by 36
  19. 2024QDyLoRA: Quantized Dynamic Low-Rank Adaptation for Efficient Large Language Model Tuningcited by 26
  20. 2024Flatten Long-Range Loss Landscapes for Cross-Domain Few-Shot Learningcited by 26
  21. 2024Do LLMs Overcome Shortcut Learning? An Evaluation of Shortcut Challenges in Large Language Modelscited by 25
  22. 2022Wave-SAN: Wavelet based Style Augmentation Network for Cross-Domain Few-Shot Learningcited by 23
  23. 2024Advancing Textual Prompt Learning with Anchored Attributescited by 20
  24. 2021Augmented Shortcuts for Vision Transformerscited by 19
  25. 2023Hydra: Multi-head Low-rank Adaptation for Parameter Efficient Fine-tuningcited by 17
  26. 2023Dual Adaptive Representation Alignment for Cross-domain Few-shot Learningcited by 15
  27. 2021Toward Learning Human-aligned Cross-domain Robust Models by Countering Misaligned Featurescited by 14
  28. 2022Rectify ViT Shortcut Learning by Visual Saliencycited by 14
  29. 2024Multimodal Cross-Domain Few-Shot Learning for Egocentric Action Recognitioncited by 14
  30. 2024Enhancing Information Maximization with Distance-Aware Contrastive Learning for Source-Free Cross-Domain Few-Shot Learningcited by 13
  31. 2025Logits DeConfusion with CLIP for Few-Shot Learningcited by 11
  32. 2024Prompt as Free Lunch: Enhancing Diversity in Source-Free Cross-domain Few-shot Learning through Semantic-Guided Promptingcited by 10
  33. 2024Shortcut Learning in In-Context Learning: A Surveycited by 8
  34. 2025Fine-Grained VLM Fine-tuning via Latent Hierarchical Adapter Learningcited by 8
  35. 2021Information Symmetry Matters: A Modal-Alternating Propagation Network for Few-Shot Learningcited by 6
  36. 2024Cross-domain Multi-modal Few-shot Object Detection via Rich Textcited by 5
  37. 2024Mind the Gap Between Prototypes and Images in Cross-domain Finetuningcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 4.

4 papers.

  1. 2026Interpretable Cross-Domain Few-Shot Learning with Rectified Target-Domain Local Alignmentcited by 4
  2. 2026Remedying Target-Domain Astigmatism for Cross-Domain Few-Shot Object Detectioncited by 2
  3. 2026Addressing Exacerbated Attention Sink for Source-Free Cross-Domain Few-Shot Learningcited by 2
  4. 2026When Semantics Saturate or Emerge: Adaptation-Conditional Semantic Utility in Source-Free Cross-Domain Few-Shot Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Reviving In-domain Fine-tuning Methods for Source-Free Cross-domain Few-shot Learning
  2. 2024VLLaVO: Mitigating Visual Gap through LLMs
  3. 2024Advancing Cross-domain Discriminability in Continual Learning of Vision-Language Models
  4. 2026Improving CLIP Adaptation by Breaking Tail Alignment for Source-Free Cross-Domain Few-Shot Learning
  5. 2025Cross-Domain Few-Shot Learning via Multi-View Collaborative Optimization with Vision-Language Models
  6. 2023Robust Fine-Tuning of Vision-Language Models for Domain Generalization
  7. 2023Few-Shot Learning with Visual Distribution Calibration and Cross-Modal Distribution Alignment
  8. 2026Conformal Cross-Modal Active Learning