mlsanityData to 7 Oct 2026

Dual Prototype Evolving for Test-Time Generalization of Vision-Language Models

arXiv 2410.12790 ↗First posted 16 Oct 2024cs.CV cs.LGcited by 36

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

Test-time adaptation, which enables models to generalize to diverse data with unlabeled test samples, holds significant value in real-world scenarios. Recently, researchers have applied this setting to advanced pre-trained vision-language models (VLMs), developing approaches such as test-time prompt tuning to further extend their practical applicability. However, these methods typically focus solely on adapting VLMs from a single modality and fail to accumulate task-specific knowledge as more samples are processed. To address this, we introduce Dual Prototype Evolving (DPE), a novel test-time adaptation approach for VLMs that effectively accumulates task-specific knowledge from multi-modalities. Specifically, we create and evolve two sets of prototypes--textual and visual--to progressively capture more accurate multi-modal representations for target classes during test time. Moreover, to promote consistent multi-modal representations, we introduce and optimize learnable residuals for each test sample to align the prototypes from both modalities. Extensive experimental results on 15 benchmark datasets demonstrate that our proposed DPE consistently outperforms previous state-of-the-art methods while also exhibiting competitive computational efficiency. Code is available at https://github.com/zhangce01/DPE-CLIP.

Key facts. Cited by 36 papers in the map (data through 7 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 69 of its 89 references that are in the map. First posted on arXiv 16 Oct 2024.

Builds on

69 of 89 references.

Closest ancestors · 18

Same direction, newest first

  1. 2024On the test-time zero-shot generalization of vision-language models: Do we really need prompt learning?3 May
  2. 2024Efficient Test-Time Adaptation of Vision-Language Models27 Mar
  3. 2024Dual Memory Networks: A Versatile Adaptation Approach for Vision-Language Models26 Mar
  4. 2024C-TPT: Calibrated Test-Time Prompt Tuning for Vision-Language Models via Text Feature Dispersion21 Mar
  5. 2024Enhancing Vision-Language Few-Shot Adaptation with Negative Learning19 Mar
  6. 2024Just Shift It: Test-Time Prototype Shifting for Zero-Shot Generalization with Vision-Language Models19 Mar
  7. 2023Align Your Prompts: Test-Time Prompting with Distribution Alignment for Zero-Shot Generalization2 Nov
  8. 2023Learning to Adapt CLIP for Few-Shot Monocular Depth Estimation2 Nov
  9. 2023Distribution-Aware Prompt Tuning for Vision-Language Models6 Sep
  10. 2023Diverse Data Augmentation with Diffusions for Effective Test-time Prompt Tuning11 Aug
  11. 2022Multitask Vision-Language Prompt Tuning21 Nov
  12. 2022Task Residual for Tuning Vision-Language Models18 Nov
6 more
  1. 2022Test-Time Prompt Tuning for Zero-Shot Generalization in Vision-Language Models15 Sep
  2. 2022Tip-Adapter: Training-free Adaption of CLIP for Few-shot Classification19 Jul
  3. 2022Prompt-aligned Gradient for Prompt Tuning30 May
  4. 2022Conditional Prompt Learning for Vision-Language Models10 Mar
  5. 2021CLIP-Adapter: Better Vision-Language Models with Feature Adapters9 Oct
  6. 2021Learning to Prompt for Vision-Language Models2 Sep

Other references · 51

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  5. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  6. 2015You Only Look Once: Unified, Real-Time Object Detectioncited by 4,829
  7. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  8. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  9. 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,857
  10. 2013Describing Textures in the Wildcited by 2,110
  11. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  12. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,842
38 more
  1. 2013Fine-Grained Visual Classification of Aircraftcited by 1,701
  2. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,507
  3. 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,488
  4. 2019Natural Adversarial Examplescited by 1,401
  5. 2020WILDS: A Benchmark of in-the-Wild Distribution Shiftscited by 1,366
  6. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,265
  7. 2022Reproducible scaling laws for contrastive language-image learningcited by 1,163
  8. 2017CyCADA: Cycle-Consistent Adversarial Domain Adaptationcited by 1,079
  9. 2019Learning Robust Global Representations by Penalizing Local Predictive Powercited by 1,010
  10. 2023Vision-Language Models for Vision Tasks: A Surveycited by 834
  11. 2022MaPLe: Multi-modal Prompt Learningcited by 685
  12. 2018Deep Visual Domain Adaptation: A Surveycited by 634
  13. 2021LiT: Zero-Shot Transfer with Locked-image text Tuningcited by 572
  14. 2023Towards Stable Test-Time Adaptation in Dynamic Wild Worldcited by 409
  15. 2021Supervision Exists Everywhere: A Data Efficient Contrastive Language-Image Pre-training Paradigmcited by 405
  16. 2021MEMO: Test Time Robustness via Adaptation and Augmentationcited by 376
  17. 2023RemoteCLIP: A Vision Language Foundation Model for Remote Sensingcited by 357
  18. 2023A Comprehensive Survey on Test-Time Adaptation under Distribution Shiftscited by 353
  19. 2021Domain Adaptation for Medical Image Analysis: A Surveycited by 351
  20. 2017"Zero-Shot" Super-Resolution using Deep Internal Learningcited by 285
  21. 2022What does a platypus look like? Generating customized prompts for zero-shot image classificationcited by 263
  22. 2022Parameter-free Online Test-time Adaptationcited by 184
  23. 2022A Survey of Vision-Language Pre-Trained Modelscited by 177
  24. 2023CORA: Adapting CLIP for Open-Vocabulary Detection with Region Prompting and Anchor Pre-Matchingcited by 133
  25. 2021The Norm Must Go On: Dynamic Unsupervised Domain Adaptation by Normalizationcited by 131
  26. 2023Multimodality Helps Unimodality: Cross-Modal Few-Shot Learning with Multimodal Modelscited by 91
  27. 2023CLIP goes 3D: Leveraging Prompt Tuning for Language Grounded 3D Recognitioncited by 77
  28. 2022MM-TTA: Multi-Modal Test-Time Adaptation for 3D Semantic Segmentationcited by 76
  29. 2019Discriminative Adversarial Domain Adaptationcited by 74
  30. 2023GraphAdapter: Tuning Vision-Language Models With Dual Knowledge Graphcited by 69
  31. 2022Can Language Understand Depth?cited by 66
  32. 2024WorDepth: Variational Language Prior for Monocular Depth Estimationcited by 34
  33. 2021Test-Time Personalization with a Transformer for Human Pose Estimationcited by 33
  34. 2021AuxAdapt: Stable and Efficient Test-Time Adaptation for Temporally Consistent Video Semantic Segmentationcited by 31
  35. 2023Neuro-Modulated Hebbian Learning for Fully Test-Time Adaptationcited by 23
  36. 2023Efficient Test-Time Adaptation for Super-Resolution with Second-Order Degradation and Reconstructioncited by 18
  37. 2024HiKER-SGG: Hierarchical Knowledge Enhanced Robust Scene Graph Generationcited by 14
  38. 2023Self-Correctable and Adaptable Inference for Generalizable Human Pose Estimationcited by 7

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 18; 2026: 18.

36 papers.

Sort:

By area:

  1. 2025Dynamic Multimodal Prototype Learning in Vision-Language Modelscited by 13
  2. 2026pySpatial: Generating 3D Visual Programs for Zero-Shot Spatial Reasoningcited by 12
  3. 2025ProtoMol: Enhancing Molecular Property Prediction via Prototype-Guided Multimodal Learningcited by 11
  4. 2025Latte: Collaborative Test-Time Adaptation of Vision-Language Models in Federated Learningcited by 9
  5. 2025Seeing What Matters: Empowering CLIP with Patch Generation-to-Selectioncited by 7
  6. 2025Bidirectional Prototype-Reward co-Evolution for Test-Time Adaptation of Vision-Language Modelscited by 6
  7. 2025Generalizing vision-language models to novel domains: A comprehensive surveycited by 6
  8. 2025Multi-Cache Enhanced Prototype Learning for Test-Time Generalization of Vision-Language Modelscited by 6
  9. 2025Backpropagation-Free Test-Time Adaptation via Probabilistic Gaussian Alignmentcited by 5
  10. 2025Mint: A Simple Test-Time Adaptation of Vision-Language Models against Common Corruptionscited by 5
  11. 2026Test-Time Adaptation via Cache Personalization for Facial Expression Recognition in Videoscited by 5
  12. 2025Negation-Aware Test-Time Adaptation for Vision-Language Modelscited by 4
24 more
  1. 2026IsoCLIP: Decomposing CLIP Projectors for Efficient Intra-modal Alignmentcited by 3
  2. 2026Multimodal Ambivalence/Hesitancy Recognition in Videos for Personalized Digital Health Interventionscited by 3
  3. 2025Mitigating Cache Noise in Test-Time Adaptation for Large Vision-Language Modelscited by 2
  4. 2025VLOD-TTA: Test-Time Adaptation of Vision-Language Object Detectorscited by 2
  5. 2025Class-Aware Prototype Learning with Negative Contrast for Test-Time Adaptation of Vision-Language Modelscited by 2
  6. 2026Test-time Ego-Exo-centric Adaptation for Action Anticipation via Multi-Label Prototype Growing and Dual-Clue Consistencycited by 2
  7. 2026CLIP-AUTT: Test-Time Personalization with Action Unit Prompting for Fine-Grained Video Emotion Recognitioncited by 2
  8. 2026Adapting in the Dark: Efficient and Stable Test-Time Adaptation for Black-Box Modelscited by 2
  9. 2026Improving Calibration in Test-Time Prompt Tuning for Vision-Language Models via Data-Free Flatness-Aware Prompt Pretrainingcited by 2
  10. 2026What Drives Test-Time Adaptation for CLIP? A Controlled Empirical Study from an Update Perspectivecited by 2
  11. 2026LENS: Adaptive Spatio-Temporal Zooming for Keyframe Sampling in Long-Form Videoscited by 2
  12. 2025Flatness Guided Test-Time Adaptation for Vision-Language Modelscited by 1
  13. 2025Uniformity First: Uniformity-aware Test-time Adaptation of Vision-language Models against Image Corruptioncited by 1
  14. 2025Unleashing the Potential of All Test Samples: Mean-Shift Guided Test-Time Adaptationcited by 1
  15. 2025Mitigating Long-Tail Bias in HOI Detection via Adaptive Diversity Cachecited by 1
  16. 2026ProtoDCS: Towards Robust and Efficient Open-Set Test-Time Adaptation for Vision-Language Modelscited by 1
  17. 2026FACTOR: Counterfactual Training-Free Test-Time Adaptation for Open-Vocabulary Object Detectioncited by 1
  18. 2026Dual Distribution Estimation for Zero-shot Noisy Test-Time Adaptation with VLMscited by 1
  19. 2026ComMem: Complementary Memory Systems for Test-Time Adaptation of Vision-Language Modelscited by 1
  20. 2025Segmentation Assisted Incremental Test Time Adaptation in an Open Worldcited by 0
  21. 2026Geometry-Preserving Unsupervised Alignment for Heterogeneous Foundation Modelscited by 0
  22. 2026One Stone, Three Birds: Self-adaptive Optimal Transport for Multi-VLM Selection, Adaptation, and Ensemblingcited by 0
  23. 2026Towards Purified Multi-Label Test-Time Adaptation of Vision-Language Modelscited by 0
  24. 2026Position Aware Layer Queries for Test Time Training in Vision Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Prototype-Based Test-Time Adaptation of Vision-Language Models
  2. 2025The Illusion of Progress? A Critical Look at Test-Time Adaptation for Vision-Language Models
  3. 2025Realistic Test-Time Adaptation of Vision-Language Models
  4. 2025Efficient Test-Time Scaling for Small Vision-Language Models
  5. 2022VLUE: A Multi-Task Benchmark for Evaluating Vision-Language Models
  6. 2025Advancing Reliable Test-Time Adaptation of Vision-Language Models under Visual Variations
  7. 2024Frustratingly Easy Test-Time Adaptation of Vision-Language Models
  8. 2025Adapting Vision-Language Models Without Labels: A Comprehensive Survey