mlsanityData to 14 Oct 2026

Distribution-Aware Prompt Tuning for Vision-Language Models

arXiv 2309.03406 ↗First posted 6 Sep 2023cs.CVcited by 35

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

Pre-trained vision-language models (VLMs) have shown impressive performance on various downstream tasks by utilizing knowledge learned from large data. In general, the performance of VLMs on target tasks can be further improved by prompt tuning, which adds context to the input image or text. By leveraging data from target tasks, various prompt-tuning methods have been studied in the literature. A key to prompt tuning is the feature space alignment between two modalities via learnable vectors with model parameters fixed. We observed that the alignment becomes more effective when embeddings of each modality are `well-arranged' in the latent space. Inspired by this observation, we proposed distribution-aware prompt tuning (DAPT) for vision-language models, which is simple yet effective. Specifically, the prompts are learned by maximizing inter-dispersion, the distance between classes, as well as minimizing the intra-dispersion measured by the distance between embeddings from the same class. Our extensive experiments on 11 benchmark datasets demonstrate that our method significantly improves generalizability. The code is available at https://github.com/mlvlab/DAPT.

Key facts. Cited by 35 papers in the map (data through 14 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 35 of its 48 references that are in the map. First posted on arXiv 6 Sep 2023.

Builds on

35 of 48 references.

Closest ancestors · 6

Same direction, newest first

  1. 2022Multitask Vision-Language Prompt Tuning21 Nov
  2. 2022Unified Vision and Language Prompt Learning13 Oct
  3. 2022PLOT: Prompt Learning with Optimal Transport for Vision-Language Models3 Oct
  4. 2022Prompt Distribution Learning6 May
  5. 2022Conditional Prompt Learning for Vision-Language Models10 Mar
  6. 2021Learning to Prompt for Vision-Language Models2 Sep

Other references · 29

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  4. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  5. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  6. 2017Prototypical Networks for Few-shot Learningcited by 3,015
  7. 2013Describing Textures in the Wildcited by 2,125
  8. 2019Language Models as Knowledge Bases?cited by 1,936
  9. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,850
  10. 2013Fine-Grained Visual Classification of Aircraftcited by 1,726
  11. 2022Visual Prompt Tuningcited by 1,681
  12. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,515
17 more
  1. 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,496
  2. 2019Natural Adversarial Examplescited by 1,408
  3. 2020Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hyperspherecited by 1,405
  4. 2019Learning Robust Global Representations by Penalizing Local Predictive Powercited by 1,013
  5. 2019How Can We Know What Language Models Know?cited by 959
  6. 2021Open-vocabulary Object Detection via Vision and Language Knowledge Distillationcited by 906
  7. 2021Florence: A New Foundation Model for Computer Visioncited by 804
  8. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 781
  9. 2022MaPLe: Multi-modal Prompt Learningcited by 687
  10. 2021FLAVA: A Foundational Language And Vision Alignment Modelcited by 649
  11. 2020Contrastive Learning of Medical Visual Representations from Paired Images and Textcited by 641
  12. 2021RegionCLIP: Region-based Language-Image Pretrainingcited by 616
  13. 2021Extract Free Dense Labels from CLIPcited by 583
  14. 2021ClipCap: CLIP Prefix for Image Captioningcited by 559
  15. 2022Learning to Prompt for Open-Vocabulary Object Detection with Vision-Language Modelcited by 317
  16. 2022CyCLIP: Cyclic Contrastive Language-Image Pretrainingcited by 127
  17. 2019KnowIT VQA: Answering Knowledge-Based Questions about Videoscited by 53

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 12; 2025: 19; 2026: 4.

35 papers.

Sort:

By area:

  1. 2024A Survey of Attacks on Large Vision-Language Models: Resources, Advances, and Future Trendscited by 84
  2. 2024Dual Prototype Evolving for Test-Time Generalization of Vision-Language Modelscited by 38
  3. 2024Boosting Vision-Language Models with Transductioncited by 36
  4. 2024EZ-HOI: VLM Adaptation via Guided Prompt Learning for Zero-Shot HOI Detectioncited by 28
  5. 2024Prompt Learning via Meta-Regularizationcited by 26
  6. 2024Adapting Multi-modal Large Language Model to Concept Drift From Pre-training Onwardscited by 19
  7. 2025A Survey on Efficient Vision-Language Modelscited by 19
  8. 2025Keeping Yourself is Important in Downstream Tuning Multimodal Large Language Modelcited by 16
  9. 2025CLIP-Powered Domain Generalization and Domain Adaptation: A Comprehensive Surveycited by 15
  10. 2025Learning to Adapt Frozen CLIP for Few-Shot Test-Time Domain Adaptationcited by 14
  11. 2024Open-Set Recognition in the Age of Vision-Language Modelscited by 13
  12. 2024Multi-modal Attribute Prompting for Vision-Language Modelscited by 12
23 more
  1. 2025Hierarchical Cross-modal Prompt Learning for Vision-Language Modelscited by 12
  2. 2025A Closer Look at Conditional Prompt Tuning for Vision-Language Modelscited by 8
  3. 2025Generalizing vision-language models to novel domains: A comprehensive surveycited by 7
  4. 2025Continual Learning on CLIP via Incremental Prompt Tuning with Intrinsic Textual Anchorscited by 6
  5. 2025Referring Expression Comprehension for Small Objectscited by 6
  6. 2024Dude: Dual Distribution-Aware Context Prompt Learning For Large Vision-Language Modelcited by 4
  7. 2025Multiplicity is an Inevitable and Inherent Challenge in Multimodal Learningcited by 4
  8. 2025Learning Generalizable Prompt for CLIP with Class Similarity Knowledgecited by 3
  9. 2025Visual Diversity and Region-aware Prompt Learning for Zero-shot HOI Detectioncited by 3
  10. 2025Diversity Covariance-Aware Prompt Learning for Vision-Language Modelscited by 2
  11. 2025Prompting without Panic: Attribute-aware, Zero-shot, Test-Time Calibrationcited by 2
  12. 2025From Query to Explanation: Uni-RAG for Multi-Modal Retrieval-Augmented Learning in STEMcited by 2
  13. 2024LPT: Less-overfitting Prompt Tuning for Vision-Language Modelcited by 1
  14. 2024Semantic Hierarchical Prompt Tuning for Parameter-Efficient Fine-Tuningcited by 1
  15. 2025Towards Fine-Grained Adaptation of CLIP via a Self-Trained Alignment Scorecited by 1
  16. 2025Beyond Graph Model: Reliable VLM Fine-Tuning via Random Graph Adaptercited by 1
  17. 2025Personalized OVSS: Understanding Personal Concept in Open-Vocabulary Semantic Segmentationcited by 1
  18. 2024Enhancing Robustness of Vision-Language Models through Orthogonality Learning and Self-Regularizationcited by 0
  19. 2025InPK: Infusing Prior Knowledge into Prompt for Vision-Language Modelscited by 0
  20. 2026Bridging the Semantic Chasm: Synergistic Conceptual Anchoring for Generalized Few-Shot and Zero-Shot OOD Perceptioncited by 0
  21. 2026VLAgeBench: Benchmarking Large Vision-Language Models for Zero-Shot Human Age Estimationcited by 0
  22. 2026P3T: Prototypical Point-level Prompt Tuning with Enhanced Generalization for 3D Vision-Language Modelscited by 0
  23. 2026Cluster-Aware Neural Collapse Prompt Tuning for Long-Tailed Generalization of Vision-Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Prompt Tuning Vision Language Models with Margin Regularizer for Few-Shot Learning under Distribution Shifts
  2. 2024Tuning Vision-Language Models with Candidate Labels by Prompt Alignment
  3. 2024Can Better Text Semantics in Prompt Tuning Improve VLM Generalization?
  4. 2023Context-Aware Prompt Tuning for Vision-Language Model with Dual-Alignment
  5. 2023Patch-Prompt Aligned Bayesian Prompt Tuning for Vision-Language Models
  6. 2024Generalizable Prompt Tuning for Vision-Language Models
  7. 2025Prompt-OT: An Optimal Transport Regularization Paradigm for Knowledge Preservation in Vision-Language Model Adaptation
  8. 2023Approximated Prompt Tuning for Vision-Language Pre-trained Models