mlsanityData to 7 Oct 2026

Black Sheep in the Herd: Playing with Spuriously Correlated Attributes for Vision-Language Recognition

arXiv 2502.15809 ↗First posted 19 Feb 2025cs.LG eess.IVcited by 7

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

Few-shot adaptation for Vision-Language Models (VLMs) presents a dilemma: balancing in-distribution accuracy with out-of-distribution generalization. Recent research has utilized low-level concepts such as visual attributes to enhance generalization. However, this study reveals that VLMs overly rely on a small subset of attributes on decision-making, which co-occur with the category but are not inherently part of it, termed spuriously correlated attributes. This biased nature of VLMs results in poor generalization. To address this, 1) we first propose Spurious Attribute Probing (SAP), identifying and filtering out these problematic attributes to significantly enhance the generalization of existing attribute-based methods; 2) We introduce Spurious Attribute Shielding (SAS), a plug-and-play module that mitigates the influence of these attributes on prediction, seamlessly integrating into various Parameter-Efficient Fine-Tuning (PEFT) methods. In experiments, SAP and SAS significantly enhance accuracy on distribution shifts across 11 datasets and 3 generalization tasks without compromising downstream performance, establishing a new state-of-the-art benchmark.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 72 of its 92 references that are in the map. First posted on arXiv 19 Feb 2025.

Builds on

72 of 92 references.

Closest ancestors · 15

Same direction, newest first

  1. 2024Dual Memory Networks: A Versatile Adaptation Approach for Vision-Language Models26 Mar
  2. 2024Calibrating Multi-modal Representations: A Pursuit of Group Robustness without Annotations12 Mar
  3. 2024Multi-modal Attribute Prompting for Vision-Language Models1 Mar
  4. 2024Concept-Guided Prompt Learning for Generalization in Vision-Language Models15 Jan
  5. 2023A Closer Look at the Few-Shot Adaptation of Large Vision-Language Models20 Dec
  6. 2023TCP:Textual-based Class-aware Prompt tuning for Visual-Language Model30 Nov
  7. 2023ArGue: Attribute-Guided Prompt Tuning for Vision-Language Models27 Nov
  8. 2023Waffling around for Performance: Visual Classification with Random Words and Broad Concepts12 Jun
  9. 2023Visual-Language Prompt Tuning with Knowledge-guided Context Optimization23 Mar
  10. 2022SuS-X: Training-Free Name-Only Transfer of Vision-Language Models28 Nov
  11. 2022Visual Classification via Description from Large Language Models13 Oct
  12. 2022Conditional Prompt Learning for Vision-Language Models10 Mar
3 more
  1. 2021Tip-Adapter: Training-free CLIP-Adapter for Better Vision-Language Modeling6 Nov
  2. 2021CLIP-Adapter: Better Vision-Language Models with Feature Adapters9 Oct
  3. 2021Learning to Prompt for Vision-Language Models2 Sep

Other references · 57

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  5. 2023Segment Anythingcited by 8,158
  6. 2023Visual Instruction Tuningcited by 7,792
  7. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  8. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  9. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  10. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  11. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  12. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
38 more
  1. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  2. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
  3. 2017The Kinetics Human Action Video Datasetcited by 1,894
  4. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,842
  5. 2013Fine-Grained Visual Classification of Aircraftcited by 1,701
  6. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,507
  7. 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,488
  8. 2019Natural Adversarial Examplescited by 1,401
  9. 2019Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalizationcited by 1,232
  10. 2021Grounded Language-Image Pre-trainingcited by 1,147
  11. 2017The "something something" video database for learning and evaluating visual common sensecited by 1,092
  12. 2020Concept Bottleneck Modelscited by 1,022
  13. 2019Learning Robust Global Representations by Penalizing Local Predictive Powercited by 1,010
  14. 2023Vision-Language Models for Vision Tasks: A Surveycited by 834
  15. 2023EVA-CLIP: Improved Training Techniques for CLIP at Scalecited by 786
  16. 2021P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Taskscited by 686
  17. 2022MaPLe: Multi-modal Prompt Learningcited by 685
  18. 2021VL-Adapter: Parameter-Efficient Transfer Learning for Vision-and-Language Taskscited by 319
  19. 2022Scaling Language-Image Pre-training via Maskingcited by 317
  20. 2022Language in a Bottle: Language Model Guided Concept Bottlenecks for Interpretable Image Classificationcited by 310
  21. 2023Self-regulating Prompts: Foundational Model Adaptation without Forgettingcited by 272
  22. 2022What does a platypus look like? Generating customized prompts for zero-shot image classificationcited by 263
  23. 2019Adversarial Domain Adaptation with Domain Mixupcited by 250
  24. 2022Improving Out-of-Distribution Robustness via Selective Augmentationcited by 202
  25. 2022Fine-tuned CLIP Models are Efficient Video Learnerscited by 178
  26. 2022Correct-N-Contrast: A Contrastive Approach for Improving Robustness to Spurious Correlationscited by 171
  27. 2020BREEDS: Benchmarks for Subpopulation Shiftcited by 155
  28. 2023Debiasing Vision-Language Models via Biased Promptscited by 147
  29. 2020Towards Debiasing NLU Models from Unknown Biasescited by 116
  30. 2021Salient ImageNet: How to discover spurious features in Deep Learning?cited by 114
  31. 2022Retrieval Augmented Classification for Long-Tail Visual Recognitioncited by 106
  32. 2022A Prompt Array Keeps the Bias Away: Debiasing Vision-Language Models with Adversarial Learningcited by 104
  33. 2021Leveraging Sparse Linear Layers for Debuggable Deep Networkscited by 82
  34. 2023An Inverse Scaling Law for CLIP Trainingcited by 76
  35. 2023Discover and Cure: Concept-aware Mitigation of Spurious Correlationcited by 71
  36. 2023DeAR: Debiasing Vision-Language Models with Additive Residualscited by 70
  37. 2024The Neglected Tails in Vision-Language Modelscited by 63
  38. 2022Contrastive Adapters for Foundation Model Group Robustnesscited by 62

50 most cited of 57 All 57 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 5; 2026: 2.

7 papers.

  1. 2025More Thought, Less Accuracy? On the Dual Nature of Reasoning in Vision-Language Modelscited by 26
  2. 2025Unlocking Vision-Language Models for Video Anomaly Detection via Fine-Grained Promptingcited by 8
  3. 2025Concept Regions Matter: Benchmarking CLIP with a New Cluster-Importance Approachcited by 2
  4. 2025High-Entropy Tokens as Multimodal Failure Points in Vision-Language Modelscited by 1
  5. 2026All Roads Lead to Rome: Incentivizing Divergent Thinking in Vision-Language Modelscited by 1
  6. 2025TARS: MinMax Token-Adaptive Preference Strategy for Hallucination Reduction in MLLMscited by 0
  7. 2026Break the Brake, Not the Wheel: Untargeted Jailbreak via Entropy Maximizationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Debiased Prompt Tuning in Vision-Language Model without Annotations
  2. 2026When Prompts Ignore Structure: Graph-Based Attribute Reasoning for Calibrated VLMs
  3. 2024RaVL: Discovering and Mitigating Spurious Correlations in Fine-Tuned Vision-Language Models
  4. 2023Vision-language Assisted Attribute Learning
  5. 2025Escaping the SpuriVerse: Can Large Vision-Language Models Generalize Beyond Seen Spurious Correlations?
  6. 2022VL-Taboo: An Analysis of Attribute-based Zero-shot Capabilities of Vision-Language Models
  7. 2026Attributes Should Come from Images, Not Class Names: Distribution-Conditioned Attribute Selection for Vision-Language Models
  8. 2025Adapting Vision-Language Models Without Labels: A Comprehensive Survey