mlsanityData to 7 Oct 2026

DPA: Dual Prototypes Alignment for Unsupervised Adaptation of Vision-Language Models

arXiv 2408.08855 ↗First posted 16 Aug 2024cs.CVcited by 5

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

Vision-language models (VLMs), e.g., CLIP, have shown remarkable potential in zero-shot image classification. However, adapting these models to new domains remains challenging, especially in unsupervised settings where labeled data is unavailable. Recent research has proposed pseudo-labeling approaches to adapt CLIP in an unsupervised manner using unlabeled target data. Nonetheless, these methods struggle due to noisy pseudo-labels resulting from the misalignment between CLIP's visual and textual representations. This study introduces DPA, an unsupervised domain adaptation method for VLMs. DPA introduces the concept of dual prototypes, acting as distinct classifiers, along with the convex combination of their outputs, thereby leading to accurate pseudo-label construction. Next, it ranks pseudo-labels to facilitate robust self-training, particularly during early training. Finally, it addresses visual-textual misalignment by aligning textual prototypes with image prototypes to further improve the adaptation performance. Experiments on 13 downstream vision tasks demonstrate that DPA significantly outperforms zero-shot CLIP and the state-of-the-art unsupervised adaptation baselines.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 49 of its 65 references that are in the map. First posted on arXiv 16 Aug 2024.

Builds on

49 of 65 references.

Closest ancestors · 9

Same direction, newest first

  1. 2023Intra-Modal Proxy Learning for Zero-Shot Visual Categorization with CLIP30 Oct
  2. 2023ReCLIP: Refine Contrastive Language Image Pre-Training with Source Free Domain Adaptation4 Aug
  3. 2023PerceptionCLIP: Visual Classification by Inferring and Conditioning on Contexts2 Aug
  4. 2022Tip-Adapter: Training-free Adaption of CLIP for Few-shot Classification19 Jul
  5. 2022Unsupervised Prompt Learning for Vision-Language Models7 Apr
  6. 2022Conditional Prompt Learning for Vision-Language Models10 Mar
  7. 2021Contrastive Vision-Language Pre-training with Limited Resources17 Dec
  8. 2021CLIP-Adapter: Better Vision-Language Models with Feature Adapters9 Oct
  9. 2021Learning to Prompt for Vision-Language Models2 Sep

Other references · 40

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  3. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  4. 2016Layer Normalizationcited by 5,579
  5. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  6. 2017Prototypical Networks for Few-shot Learningcited by 3,001
  7. 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,261
  8. 2013Describing Textures in the Wildcited by 2,110
  9. 2020FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidencecited by 2,080
  10. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,842
  11. 2018Unsupervised Feature Learning via Non-Parametric Instance-level Discriminationcited by 1,838
  12. 2013Fine-Grained Visual Classification of Aircraftcited by 1,701
28 more
  1. 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,488
  2. 2019Natural Adversarial Examplescited by 1,401
  3. 2019Learning Robust Global Representations by Penalizing Local Predictive Powercited by 1,010
  4. 2022MaPLe: Multi-modal Prompt Learningcited by 685
  5. 2021LiT: Zero-Shot Transfer with Locked-image text Tuningcited by 572
  6. 2022Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learningcited by 553
  7. 2021SLIP: Self-supervision meets Language-Image Pre-trainingcited by 434
  8. 2021Supervision Exists Everywhere: A Data Efficient Contrastive Language-Image Pre-training Paradigmcited by 405
  9. 2020VirTex: Learning Visual Representations from Textual Annotationscited by 338
  10. 2019Label Propagation for Deep Semi-supervised Learningcited by 287
  11. 2023Self-regulating Prompts: Foundational Model Adaptation without Forgettingcited by 272
  12. 2017Zero-Shot Learning -- The Good, the Bad and the Uglycited by 254
  13. 2022Vision-Language Pre-Training with Triple Contrastive Learningcited by 212
  14. 2021Scaling Up Vision-Language Pre-training for Image Captioningcited by 196
  15. 2020CoMatch: Semi-supervised Learning with Contrastive Graph Regularizationcited by 167
  16. 2020Attribute Prototype Network for Zero-Shot Learningcited by 160
  17. 2023Prompt, Generate, then Cache: Cascade of Foundation Models makes Strong Few-shot Learnerscited by 152
  18. 2020Learning Visual Representations with Caption Annotationscited by 123
  19. 2018Robust Classification with Convolutional Prototype Learningcited by 122
  20. 2019Hyperspherical Prototype Networkscited by 85
  21. 2022Debiased Learning from Naturally Imbalanced Pseudo-Labelscited by 85
  22. 2023Not All Features Matter: Enhancing Few-shot CLIP with Adaptive Prior Refinementcited by 84
  23. 2022Parameter-efficient Model Adaptation for Vision Transformerscited by 81
  24. 2023LaFTer: Label-Free Tuning of Zero-shot Classifier using Language and Unlabeled Image Collectionscited by 43
  25. 2023POUF: Prompt-oriented unsupervised fine-tuning for large pre-trained modelscited by 29
  26. 2022Few-shot Learning with Noisy Labelscited by 23
  27. 2022Masked Unsupervised Self-training for Label-free Image Classificationcited by 14
  28. 2021Better Self-training for Image Classification through Self-supervisioncited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 4; 2026: 1.

5 papers.

  1. 2025CLIP-Powered Domain Generalization and Domain Adaptation: A Comprehensive Surveycited by 15
  2. 2025Multi-Cache Enhanced Prototype Learning for Test-Time Generalization of Vision-Language Modelscited by 6
  3. 2025Test-Time Consistency in Vision Language Modelscited by 5
  4. 2025Towards Fine-Grained Adaptation of CLIP via a Self-Trained Alignment Scorecited by 1
  5. 2026Dual-Stream Semantic Guidance with Prototype Anchor Calibration for Source-Fully-Free Adaptation of Vision-Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Prototype-Guided Pseudo-Labeling with Neighborhood-Aware Consistency for Unsupervised Adaptation
  2. 2025Adapting Vision-Language Models Without Labels: A Comprehensive Survey
  3. 2023Unsupervised Prototype Adapter for Vision-Language Models
  4. 2026Supervised Classification Heads as Semantic Prototypes: Unlocking Vision-Language Alignment via Weight Recycling
  5. 2025Training-Free Label Space Alignment for Universal Domain Adaptation
  6. 2025Multi-Prompt Progressive Alignment for Multi-Source Unsupervised Domain Adaptation
  7. 2024Learning to Rank Pre-trained Vision-Language Models for Downstream Tasks
  8. 2026From Inference to Adaptation: A Unified Optimal Transport View of Vision Language Model