mlsanityData to 7 Oct 2026

ArGue: Attribute-Guided Prompt Tuning for Vision-Language Models

arXiv 2311.16494 ↗First posted 27 Nov 2023cs.CVcited by 39

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

Although soft prompt tuning is effective in efficiently adapting Vision-Language (V&L) models for downstream tasks, it shows limitations in dealing with distribution shifts. We address this issue with Attribute-Guided Prompt Tuning (ArGue), making three key contributions. 1) In contrast to the conventional approach of directly appending soft prompts preceding class names, we align the model with primitive visual attributes generated by Large Language Models (LLMs). We posit that a model's ability to express high confidence in these attributes signifies its capacity to discern the correct class rationales. 2) We introduce attribute sampling to eliminate disadvantageous attributes, thus only semantically meaningful attributes are preserved. 3) We propose negative prompting, explicitly enumerating class-agnostic attributes to activate spurious correlations and encourage the model to generate highly orthogonal probability distributions in relation to these negative features. In experiments, our method significantly outperforms current state-of-the-art prompt tuning methods on both novel class prediction and out-of-distribution generalization tasks.

Key facts. Cited by 39 papers in the map (data through 7 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 38 of its 52 references that are in the map. First posted on arXiv 27 Nov 2023.

Builds on

38 of 52 references.

Closest ancestors · 9

Same direction, newest first

  1. 2023Tuning Multi-mode Token-level Prompt Alignment across Modalities25 Sep
  2. 2023Learning Concise and Descriptive Attributes for Visual Recognition7 Aug
  3. 2023Waffling around for Performance: Visual Classification with Random Words and Broad Concepts12 Jun
  4. 2023Patch-Prompt Aligned Bayesian Prompt Tuning for Vision-Language Models16 Mar
  5. 2023Debiased Fine-Tuning for Vision-language Models by Prompt Regularization29 Jan
  6. 2022Visual Classification via Description from Large Language Models13 Oct
  7. 2022Prompt Distribution Learning6 May
  8. 2022Conditional Prompt Learning for Vision-Language Models10 Mar
  9. 2021Learning to Prompt for Vision-Language Models2 Sep

Other references · 29

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,690
  6. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  7. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  8. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  9. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  10. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
  11. 2013Describing Textures in the Wildcited by 2,110
  12. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,842
17 more
  1. 2013Fine-Grained Visual Classification of Aircraftcited by 1,701
  2. 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,488
  3. 2019Natural Adversarial Examplescited by 1,401
  4. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,255
  5. 2019Learning Robust Global Representations by Penalizing Local Predictive Powercited by 1,010
  6. 2019How Can We Know What Language Models Know?cited by 953
  7. 2019Universal Adversarial Triggers for Attacking and Analyzing NLPcited by 709
  8. 2022MaPLe: Multi-modal Prompt Learningcited by 685
  9. 2022Language in a Bottle: Language Model Guided Concept Bottlenecks for Interpretable Image Classificationcited by 310
  10. 2022What does a platypus look like? Generating customized prompts for zero-shot image classificationcited by 263
  11. 2021SPoT: Better Frozen Model Adaptation through Soft Prompt Transfercited by 235
  12. 2019Commonsense Knowledge Mining from Pretrained Modelscited by 189
  13. 2021BERTese: Learning to Speak to BERTcited by 82
  14. 2022LASP: Text-to-Text Optimization for Language-Aware Soft Prompting of Vision & Language Modelscited by 58
  15. 2023Exposing and Mitigating Spurious Correlations for Cross-Modal Retrievalcited by 35
  16. 2022Reducing Retraining by Recycling Parameter-Efficient Promptscited by 11
  17. 2023Training with Product Digital Twins for AutoRetail Checkoutcited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 10; 2025: 23; 2026: 6.

39 papers.

Sort:

By area:

  1. 2025More Thought, Less Accuracy? On the Dual Nature of Reasoning in Vision-Language Modelscited by 26
  2. 2024Advancing Textual Prompt Learning with Anchored Attributescited by 20
  3. 2025DPC: Dual-Prompt Collaboration for Tuning Vision-Language Modelscited by 17
  4. 2025Keeping Yourself is Important in Downstream Tuning Multimodal Large Language Modelcited by 15
  5. 2024Rethinking Misalignment in Vision-Language Model Adaptation from a Causal Perspectivecited by 9
  6. 2024Enhancing Vision-Language Few-Shot Adaptation with Negative Learningcited by 8
  7. 2025Unlocking Vision-Language Models for Video Anomaly Detection via Fine-Grained Promptingcited by 8
  8. 2024Tree of Attributes Prompt Learning for Vision-Language Modelscited by 7
  9. 2025Attribute-based Visual Reprogramming for Vision-Language Modelscited by 7
  10. 2025Black Sheep in the Herd: Playing with Spuriously Correlated Attributes for Vision-Language Recognitioncited by 7
  11. 2025OSLoPrompt: Bridging Low-Supervision Challenges and Open-Set Domain Generalization in CLIPcited by 6
  12. 2025Interpretable Zero-Shot Learning with Locally-Aligned Vision-Language Modelcited by 6
27 more
  1. 2026MMLoP: Multi-Modal Low-Rank Prompting for Efficient Vision-Language Adaptationcited by 5
  2. 2024Aggregate-and-Adapt Natural Language Prompts for Downstream Generalization of CLIPcited by 4
  3. 2025MMRL++: Parameter-Efficient and Interaction-Aware Representation Learning for Vision-Language Modelscited by 4
  4. 2025HERMAN: Hierarchical Representation Matching for CLIP-based Class-Incremental Learningcited by 3
  5. 2025VaMP: Variational Multi-Modal Prompt Learning for Vision-Language Modelscited by 3
  6. 2026A Visual Semantic Adaptive Watermark grounded by Prefix-Tuning for Large Vision-Language Modelcited by 3
  7. 2024Real Classification by Description: Extending CLIP's Limits of Part Attributes Recognitioncited by 2
  8. 2025Prompting without Panic: Attribute-aware, Zero-shot, Test-Time Calibrationcited by 2
  9. 2024CoAPT: Context Attribute words for Prompt Tuningcited by 1
  10. 2024Craft: Cross-modal Aligned Features Improve Robustness of Prompt Tuningcited by 1
  11. 2024Semantic Hierarchical Prompt Tuning for Parameter-Efficient Fine-Tuningcited by 1
  12. 2025Generalizable Vision-Language Few-Shot Adaptation with Predictive Prompts and Negative Learningcited by 1
  13. 2025FDBPL: Faster Distillation-Based Prompt Learning for Region-Aware Vision-Language Models Adaptationcited by 1
  14. 2025Proxy-FDA: Proxy-based Feature Distribution Alignment for Fine-tuning Vision Foundation Models without Forgettingcited by 1
  15. 2025AnchorOPT: Towards Optimizing Dynamic Anchors for Adaptive Prompt Learningcited by 1
  16. 2026Mind the Way You Select Negative Texts: Pursuing the Distance Consistency in OOD Detection with VLMscited by 1
  17. 2026FVG-PT: Adaptive Foreground View-Guided Prompt Tuning for Vision-Language Modelscited by 1
  18. 2024LPT: Less-overfitting Prompt Tuning for Vision-Language Modelcited by 0
  19. 2025TAIL: Text-Audio Incremental Learningcited by 0
  20. 2025Dual-Path Stable Soft Prompt Generation for Domain Generalizationcited by 0
  21. 2025The Devil is in the Darkness: Diffusion-Based Nighttime Dehazing Anchored in Brightness Perceptioncited by 0
  22. 2025Exploiting Vision Language Model for Training-Free 3D Point Cloud OOD Detection via Graph Score Propagationcited by 0
  23. 2025SemPT: Semantic Prompt Tuning for Vision-Language Modelscited by 0
  24. 2025GA2-CLIP: Generic Attribute Anchor for Efficient Prompt Tuningin Video-Language Modelscited by 0
  25. 2025AmPLe: Supporting Vision-Language Models via Adaptive-Debiased Ensemble Multi-Prompt Learningcited by 0
  26. 2026Break the Brake, Not the Wheel: Untargeted Jailbreak via Entropy Maximizationcited by 0
  27. 2026Grad-CAM for Vision Transformers: A Systematic Taxonomy and Audit of Methodological Ambiguity in Explainable AIcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025A Closer Look at Conditional Prompt Tuning for Vision-Language Models
  2. 2024Generalizable Prompt Tuning for Vision-Language Models
  3. 2024Can Better Text Semantics in Prompt Tuning Improve VLM Generalization?
  4. 2023Towards Efficient Vision-Language Tuning: More Information Density, More Generalizability
  5. 2024Multi-modal Attribute Prompting for Vision-Language Models
  6. 2022Test-Time Prompt Tuning for Zero-Shot Generalization in Vision-Language Models
  7. 2024Adapting Vision-Language Models to Open Classes via Test-Time Prompt Tuning
  8. 2024Nemesis: Normalizing the Soft-prompt Vectors of Vision-Language Models