mlsanityData to 7 Oct 2026

When Adaptation Fails: A Gradient-Based Diagnosis of Collapsed Gating in Vision-Language Prompt Learning

arXiv 2605.09549 ↗First posted 10 May 2026cs.LGcited by 0

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

Adaptive prompting mechanisms have been proposed to enhance vision-language models by dynamically tailoring prompts to inputs. However, in frozen few-shot prompt learning with CLIP-style backbones, we systematically observe that adaptive gates and prompt-selection modules often collapse: they produce nearly constant outputs, contribute negligible gradient signals, and frequently fail to outperform fixed prompts. To further explore this issue, we present a systematic diagnostic study to uncover the underlying causes and conditions of adaptation failure. Through controlled experiments across datasets and multiple prompt learning architectures, we identify two recurring failure modes: gradient magnitude imbalance and gate degradation. Our findings invite a re-examination of indiscriminately adding architectural complexity in parameter-efficient learning and clarify when prompt-level adaptive gating is, and is not, effective in this regime.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 17 of its 22 references that are in the map. First posted on arXiv 10 May 2026.

Builds on

17 of 22 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024Vision-Language Model Fine-Tuning via Simple Parameter-Efficient Modification25 Sep
  2. 2022Conditional Prompt Learning for Vision-Language Models10 Mar
  3. 2021CLIP-Adapter: Better Vision-Language Models with Feature Adapters9 Oct
  4. 2021Learning to Prompt for Vision-Language Models2 Sep

Other references · 13

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2022Visual Prompt Tuningcited by 1,674
  4. 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,488
  5. 2022MaPLe: Multi-modal Prompt Learningcited by 685
  6. 2018Soft Filter Pruning for Accelerating Deep Convolutional Neural Networkscited by 313
  7. 2020Gradient Starvation: A Learning Proclivity in Neural Networkscited by 262
  8. 2023Stabilizing Transformer Training by Preventing Attention Entropy Collapsecited by 154
  9. 2022Signal Propagation in Transformers: Theoretical Perspectives and the Role of Rank Collapsecited by 136
  10. 2024ATP-LLaVA: Adaptive Token Pruning for Large Vision Language Modelscited by 89
  11. 2023Adaptive Gating in Mixture-of-Experts based Language Modelscited by 16
  12. 2024DEGAP: Dual Event-Guided Adaptive Prefixes for Templated-Based Event Argument Extraction with Slot Queryingcited by 3
1 more
  1. 2025PromptHash: Affinity-Prompted Collaborative Cross-Modal Learning for Adaptive Hashing Retrievalcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025IAP: Improving Continual Learning of Vision-Language Models via Instance-Aware Prompting
  2. 2026Guided Prompt Evolution for Vision-Language Models Adaptation
  3. 2026Towards Fine-Grained Robustness: Attention-Guided Test-Time Prompt Tuning for Vision-Language Models
  4. 2026Text Capability Loss in Vision-Language Adaptation: An Attention-Sink Diagnosis
  5. 2023Gradient-Regulated Meta-Prompt Learning for Generalizable Vision-Language Models
  6. 2025A Closer Look at Conditional Prompt Tuning for Vision-Language Models
  7. 2026Improving Calibration in Test-Time Prompt Tuning for Vision-Language Models via Data-Free Flatness-Aware Prompt Pretraining
  8. 2024A Lost Opportunity for Vision-Language Models: A Comparative Study of Online Test-Time Adaptation for Vision-Language Models