mlsanityData to 14 Oct 2026

Enhancing Zero-Shot Vision Models by Label-Free Prompt Distribution Learning and Bias Correcting

arXiv 2410.19294 ↗First posted 25 Oct 2024cs.CVcited by 26

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

Vision-language models, such as CLIP, have shown impressive generalization capacities when using appropriate text descriptions. While optimizing prompts on downstream labeled data has proven effective in improving performance, these methods entail labor costs for annotations and are limited by their quality. Additionally, since CLIP is pre-trained on highly imbalanced Web-scale data, it suffers from inherent label bias that leads to suboptimal performance. To tackle the above challenges, we propose a label-Free prompt distribution learning and bias correction framework, dubbed as **Frolic**, which boosts zero-shot performance without the need for labeled data. Specifically, our Frolic learns distributions over prompt prototypes to capture diverse visual representations and adaptively fuses these with the original CLIP through confidence matching. This fused model is further enhanced by correcting label bias via a label-free logit adjustment. Notably, our method is not only training-free but also circumvents the necessity for hyper-parameter tuning. Extensive experimental results across 16 datasets demonstrate the efficacy of our approach, particularly outperforming the state-of-the-art by an average of 2.6% on 10 datasets with CLIP ViT-B/16 and achieving an average margin of 1.5% on ImageNet and its five distribution shifts with CLIP ViT-B/16. Codes are available in https://github.com/zhuhsingyuu/Frolic.

Key facts. Cited by 26 papers in the map (data through 14 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 39 of its 51 references that are in the map. First posted on arXiv 25 Oct 2024.

Builds on

39 of 51 references.

Closest ancestors · 12

Same direction, newest first

  1. 2024Selective Vision-Language Subspace Projection for Few-shot CLIP24 Jul
  2. 2024Efficient Test-Time Adaptation of Vision-Language Models27 Mar
  3. 2023Align Your Prompts: Test-Time Prompting with Distribution Alignment for Zero-Shot Generalization2 Nov
  4. 2023Intra-Modal Proxy Learning for Zero-Shot Visual Categorization with CLIP30 Oct
  5. 2023Black Box Few-Shot Adaptation for Vision-Language models4 Apr
  6. 2022SuS-X: Training-Free Name-Only Transfer of Vision-Language Models28 Nov
  7. 2022Test-Time Prompt Tuning for Zero-Shot Generalization in Vision-Language Models15 Sep
  8. 2022Tip-Adapter: Training-free Adaption of CLIP for Few-shot Classification19 Jul
  9. 2022Prompt-aligned Gradient for Prompt Tuning30 May
  10. 2022Prompt Distribution Learning6 May
  11. 2022Conditional Prompt Learning for Vision-Language Models10 Mar
  12. 2021Learning to Prompt for Vision-Language Models2 Sep

Other references · 27

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2017On Calibration of Modern Neural Networkscited by 4,960
  3. 2013Describing Textures in the Wildcited by 2,125
  4. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,850
  5. 2013Fine-Grained Visual Classification of Aircraftcited by 1,726
  6. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,515
  7. 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,496
  8. 2019Natural Adversarial Examplescited by 1,408
  9. 2022Reproducible scaling laws for contrastive language-image learningcited by 1,173
  10. 2019Learning Robust Global Representations by Penalizing Local Predictive Powercited by 1,013
  11. 2021Robust fine-tuning of zero-shot modelscited by 797
  12. 2020Long-tail learning via logit adjustmentcited by 618
15 more
  1. 2022What does a platypus look like? Generating customized prompts for zero-shot image classificationcited by 267
  2. 2020Long-Tailed Classification by Keeping the Good and Removing the Bad Momentum Causal Effectcited by 256
  3. 2020Disentangling Label Distribution for Long-tailed Visual Recognitioncited by 172
  4. 2022Debiased Learning from Naturally Imbalanced Pseudo-Labelscited by 85
  5. 2024The Neglected Tails in Vision-Language Modelscited by 63
  6. 2024A Hard-to-Beat Baseline for Training-free CLIP-based Adaptationcited by 53
  7. 2023A Simple Zero-shot Prompt Weighting Technique to Improve Prompt Ensembling in Text-Image Modelscited by 39
  8. 2023GPT4Vis: What Can GPT-4 Do for Zero-shot Visual Recognition?cited by 35
  9. 2022Calibrated ensembles can mitigate accuracy tradeoffs under distribution shiftcited by 34
  10. 2023Generalized Logit Adjustment: Calibrating Fine-tuned Models by Removing Label Bias in Foundation Modelscited by 32
  11. 2021Cross-Domain Empirical Risk Minimization for Unbiased Long-tailed Classificationcited by 19
  12. 2024On Catastrophic Inheritance of Large Foundation Modelscited by 15
  13. 2023Invariant Training 2D-3D Joint Hard Samples for Few-Shot Point Cloud Recognitioncited by 13
  14. 2023Bi-directional Distribution Alignment for Transductive Zero-Shot Learningcited by 7
  15. 2020Privacy Preserving Recalibration under Domain Shiftcited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 13; 2026: 12.

26 papers.

Sort:

By area:

  1. 2025On the Generalization of SFT: A Reinforcement Learning Perspective with Reward Rectificationcited by 127
  2. 2025Enhancing CLIP Robustness via Cross-Modality Alignmentcited by 15
  3. 2025Accelerating Diffusion Transformer via Error-Optimized Cachecited by 14
  4. 2025Dynamic Multimodal Prototype Learning in Vision-Language Modelscited by 13
  5. 2025Real-Time Motion-Controllable Autoregressive Video Diffusioncited by 9
  6. 2026CatRAG: Functor-Guided Structural Debiasing with Retrieval Augmentation for Fair LLMscited by 9
  7. 2026Look Carefully: Adaptive Visual Reinforcements in Multimodal Large Language Models for Hallucination Mitigationcited by 7
  8. 2025ACE: Concept Editing in Diffusion Models without Performance Degradationcited by 6
  9. 2025Backpropagation-Free Test-Time Adaptation via Probabilistic Gaussian Alignmentcited by 6
  10. 2025Hierarchical Semantic Alignment for Image Clusteringcited by 5
  11. 2024Context-Aware Multimodal Pretrainingcited by 3
  12. 2025ASAP: Unsupervised Post-training with Label Distribution Shift Adaptive Learning Ratecited by 3
14 more
  1. 2025Visual Diversity and Region-aware Prompt Learning for Zero-shot HOI Detectioncited by 3
  2. 2026Adaptive Debiasing Tsallis Entropy for Test-Time Adaptationcited by 3
  3. 2025OASIS: Open-world Adaptive Self-supervised and Imbalanced-aware Systemcited by 2
  4. 2025Causality-guided Prompt Learning for Vision-language Models via Visual Granulationcited by 2
  5. 2026Principled Steering via Null-space Projection for Jailbreak Defense in Vision-Language Modelscited by 2
  6. 2026Dual Distribution Estimation for Zero-shot Noisy Test-Time Adaptation with VLMscited by 2
  7. 2025TOFA: Training-Free One-Shot Federated Adaptation for Vision-Language Modelscited by 1
  8. 2026Adapting Point Cloud Analysis via Multimodal Bayesian Distribution Learningcited by 0
  9. 2026[CLS] is Not Enough: Multi-Label Recognition via Patch-Level Inference and Adaptive Aggregationcited by 0
  10. 2026The Geometry of Saturation: Effective Rank Predicts When Labels Stop Helping in Few-Shot Classificationcited by 0
  11. 2026Dynamic Distribution-Aware Uncertainty Tracking in Vision-Language Representation Learningcited by 0
  12. 2026PEA-DPO: Perception-Enhanced Alignment Direct Preference Optimization for MLLMs Alignmentcited by 0
  13. 2026Membership Inference in Fine-tuned Diffusion Language Models via Token-level Memorization Asymmetrycited by 0
  14. 2026Domain Recentering and Confidence-Weighted Prior Calibration for Vision-Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Robust Fine-Tuning of Vision-Language Models for Domain Generalization
  2. 2026ZOMP: Zeroth-Order Multi-Modal Prompt Tuning for Vision-Language Models
  3. 2024On the test-time zero-shot generalization of vision-language models: Do we really need prompt learning?
  4. 2023Improving Zero-Shot Generalization for CLIP with Synthesized Prompts
  5. 2022VL-Taboo: An Analysis of Attribute-based Zero-shot Capabilities of Vision-Language Models
  6. 2026Label Shift Aware Adaptation for Online Zero-shot Learning with Contrastive Language-Image Pre-Training (CLIP)
  7. 2023Enhancing CLIP with CLIP: Exploring Pseudolabeling for Limited-Label Prompt Tuning
  8. 2025Prompt Tuning Vision Language Models with Margin Regularizer for Few-Shot Learning under Distribution Shifts