mlsanityData to 7 Oct 2026

Just Shift It: Test-Time Prototype Shifting for Zero-Shot Generalization with Vision-Language Models

arXiv 2403.12952 ↗First posted 19 Mar 2024cs.CV cs.AI cs.LGcited by 31

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

Advancements in vision-language models (VLMs) have propelled the field of computer vision, particularly in the zero-shot learning setting. Despite their promise, the effectiveness of these models often diminishes due to domain shifts in test environments. To address this, we introduce the Test-Time Prototype Shifting (TPS) framework, a pioneering approach designed to adapt VLMs to test datasets using unlabeled test inputs. Our method is based on the notion of modulating per-class prototypes in the shared embedding space. By pre-computing and caching prototypes generated with the pre-trained text encoder, TPS not only facilitates optimization-free prototype reuse for subsequent predictions but also enables seamless integration with current advancements in prompt engineering. At test-time, TPS dynamically learns shift vectors for each prototype based solely on the given test sample, effectively bridging the domain gap and enhancing classification accuracy. A notable aspect of our framework is its significantly reduced memory and computational demands when compared to conventional text-prompt tuning methods. Extensive evaluations across 15 image classification datasets involving natural distribution shifts and cross-dataset generalization, as well as in context-dependent visual reasoning, demonstrate TPS's superior performance, achieving state-of-the-art results while reducing resource requirements.

Key facts. Cited by 31 papers in the map (data through 7 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 43 of its 64 references that are in the map. First posted on arXiv 19 Mar 2024.

Builds on

43 of 64 references.

Closest ancestors · 11

Same direction, newest first

  1. 2023Align Your Prompts: Test-Time Prompting with Distribution Alignment for Zero-Shot Generalization2 Nov
  2. 2023Language Models as Black-Box Optimizers for Vision-Language Models12 Sep
  3. 2023Diverse Data Augmentation with Diffusions for Effective Test-time Prompt Tuning11 Aug
  4. 2023Test-Time Adaptation with CLIP Reward for Zero-Shot Generalization in Vision-Language Models29 May
  5. 2023Test-Time Distribution Normalization for Contrastively Learned Vision-language Models22 Feb
  6. 2022SuS-X: Training-Free Name-Only Transfer of Vision-Language Models28 Nov
  7. 2022Visual Classification via Description from Large Language Models13 Oct
  8. 2022Test-Time Prompt Tuning for Zero-Shot Generalization in Vision-Language Models15 Sep
  9. 2022Conditional Prompt Learning for Vision-Language Models10 Mar
  10. 2021CLIP-Adapter: Better Vision-Language Models with Feature Adapters9 Oct
  11. 2021Learning to Prompt for Vision-Language Models2 Sep

Other references · 32

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  4. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  5. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  6. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  7. 2017FiLM: Visual Reasoning with a General Conditioning Layercited by 2,286
  8. 2013Describing Textures in the Wildcited by 2,110
  9. 2017Arbitrary Style Transfer in Real-time with Adaptive Instance Normalizationcited by 1,998
  10. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,842
  11. 2013Fine-Grained Visual Classification of Aircraftcited by 1,701
  12. 2022Visual Prompt Tuningcited by 1,674
20 more
  1. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,507
  2. 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,488
  3. 2015Spatial Transformer Networkscited by 1,423
  4. 2019Natural Adversarial Examplescited by 1,401
  5. 2023ImageBind: One Embedding Space To Bind Them Allcited by 1,162
  6. 2019Learning Robust Global Representations by Penalizing Local Predictive Powercited by 1,010
  7. 2022Efficient Test-Time Model Adaptation without Forgettingcited by 467
  8. 2020Improving robustness against common corruptions by covariate shift adaptationcited by 427
  9. 2023Towards Stable Test-Time Adaptation in Dynamic Wild Worldcited by 409
  10. 2021MEMO: Test Time Robustness via Adaptation and Augmentationcited by 376
  11. 2022Language in a Bottle: Language Model Guided Concept Bottlenecks for Interpretable Image Classificationcited by 310
  12. 2022Scaling & Shifting Your Features: A New Baseline for Efficient Model Tuningcited by 292
  13. 2017Modulating early visual processing by languagecited by 167
  14. 2021End-to-End Human Object Interaction Detection with HOI Transformercited by 134
  15. 2023EcoTTA: Memory-Efficient Continual Test-time Adaptation via Self-distilled Regularizationcited by 123
  16. 2022CALIP: Zero-Shot Enhancement of CLIP with Parameter-free Attentioncited by 103
  17. 2022Test-Time Adaptation via Conjugate Pseudo-labelscited by 98
  18. 2022Video Test-Time Adaptation for Action Recognitioncited by 35
  19. 2022Bongard-HOI: Benchmarking Few-Shot Visual Reasoning for Human-Object Interactionscited by 34
  20. 2018On Self Modulation for Generative Adversarial Networkscited by 33

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 17; 2026: 11.

31 papers.

Sort:

By area:

  1. 2024Frustratingly Easy Test-Time Adaptation of Vision-Language Modelscited by 36
  2. 2024Dual Prototype Evolving for Test-Time Generalization of Vision-Language Modelscited by 36
  3. 2025R-TPT: Improving Adversarial Robustness of Vision-Language Models through Test-Time Prompt Tuningcited by 33
  4. 2025Latte: Collaborative Test-Time Adaptation of Vision-Language Models in Federated Learningcited by 9
  5. 2025Free on the Fly: Enhancing Flexibility in Test-Time Adaptation with Online EMcited by 8
  6. 2025Bidirectional Prototype-Reward co-Evolution for Test-Time Adaptation of Vision-Language Modelscited by 6
  7. 2025Multi-Cache Enhanced Prototype Learning for Test-Time Generalization of Vision-Language Modelscited by 6
  8. 2025Backpropagation-Free Test-Time Adaptation via Probabilistic Gaussian Alignmentcited by 5
  9. 2025Mint: A Simple Test-Time Adaptation of Vision-Language Models against Common Corruptionscited by 5
  10. 2025Self-Calibrated Consistency can Fight Back for Adversarial Robustness in Vision-Language Modelscited by 3
  11. 2025Test-Time Spectrum-Aware Latent Steering for Zero-Shot Generalization in Vision-Language Modelscited by 3
  12. 2025Mitigating Cache Noise in Test-Time Adaptation for Large Vision-Language Modelscited by 2
19 more
  1. 2025Class-Aware Prototype Learning with Negative Contrast for Test-Time Adaptation of Vision-Language Modelscited by 2
  2. 2025TTP: Test-Time Padding for Adversarial Detection and Robust Adaptation on Vision-Language Modelscited by 2
  3. 2026What Drives Test-Time Adaptation for CLIP? A Controlled Empirical Study from an Update Perspectivecited by 2
  4. 2024S3: Synonymous Semantic Space for Improving Zero-Shot Generalization of Vision-Language Modelscited by 1
  5. 2025Uniformity First: Uniformity-aware Test-time Adaptation of Vision-language Models against Image Corruptioncited by 1
  6. 2025Constrained Prompt Enhancement for Improving Zero-Shot Generalization of Vision-Language Modelscited by 1
  7. 2026Dual Distribution Estimation for Zero-shot Noisy Test-Time Adaptation with VLMscited by 1
  8. 2026ComMem: Complementary Memory Systems for Test-Time Adaptation of Vision-Language Modelscited by 1
  9. 2025Hierarchical Adaptive networks with Task vectors for Test-Time Adaptationcited by 0
  10. 2025TRANSPORTER: Transferring Visual Semantics from VLM Manifoldscited by 0
  11. 2025Measurement Plasticity: Sensor-Level Adaptation for Vision-Language Modelscited by 0
  12. 2026Fair Context Learning for Evidence-Balanced Test-Time Adaptation in Vision-Language Modelscited by 0
  13. 2026What Makes VLMs Robust? Towards Reconciling Robustness and Accuracy in Vision-Language Modelscited by 0
  14. 2026Ramen: Robust Test-Time Adaptation of Vision-Language Models with Active Sample Selectioncited by 0
  15. 2026AGC: Adaptive Geodesic Correction for Adversarial Robustness on Vision-Language Modelscited by 0
  16. 2026Drift-Augmented Scoring: Text-Derived Noise Robustness for Zero-Shot Audio-Language Classificationcited by 0
  17. 2026Cross-device Collaborative Test-time Adaptation with Zeroth-order Optimization and Model Mergingcited by 0
  18. 2026Bridging the Confidence Gap: Temperature Scaling for Calibrating Test-Time Prompt Tuningcited by 0
  19. 2026Position Aware Layer Queries for Test Time Training in Vision Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026ProtoDCS: Towards Robust and Efficient Open-Set Test-Time Adaptation for Vision-Language Models
  2. 2025Dynamic Multimodal Prototype Learning in Vision-Language Models
  3. 2024PromptSync: Bridging Domain Gaps in Vision-Language Models through Class-Aware Prototype Alignment and Discrimination
  4. 2025Ultra-Light Test-Time Adaptation for Vision--Language Models
  5. 2026Efficient Test-time Adaptation through Candidate Verification and Divergence Shifts
  6. 2024BaFTA: Backprop-Free Test-Time Adaptation For Zero-Shot Vision-Language Models
  7. 2026Prototype-Based Test-Time Adaptation of Vision-Language Models
  8. 2025Realistic Test-Time Adaptation of Vision-Language Models