# Bayesian Prompt Learning for Image-Language Model Generalization

arXiv: https://arxiv.org/abs/2210.02390 · HTML: https://mlsanity.com/paper/2210.02390/

## Key facts

- Cited by 51 papers in the map (data through 14 Oct 2026).
- Builds on 45 of its 59 references that are in the map.
- First posted on arXiv 5 Oct 2022.

## Builds on (most cited first, 30 of 45)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [Flamingo: a Visual Language Model for Few-Shot Learning](https://mlsanity.com/paper/2204.14198/) (2022)
- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [Prefix-Tuning: Optimizing Continuous Prompts for Generation](https://mlsanity.com/paper/2101.00190/) (2021)
- [Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision](https://mlsanity.com/paper/2102.05918/) (2021)
- [The Power of Scale for Parameter-Efficient Prompt Tuning](https://mlsanity.com/paper/2104.08691/) (2021)
- [Learning to Prompt for Vision-Language Models](https://mlsanity.com/paper/2109.01134/) (2021)
- [Describing Textures in the Wild](https://mlsanity.com/paper/1311.3618/) (2013)
- [Invariant Risk Minimization](https://mlsanity.com/paper/1907.02893/) (2019)
- [The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalization](https://mlsanity.com/paper/2006.16241/) (2020)
- [Fine-Grained Visual Classification of Aircraft](https://mlsanity.com/paper/1306.5151/) (2013)
- [Visual Prompt Tuning](https://mlsanity.com/paper/2203.12119/) (2022)
- [Do ImageNet Classifiers Generalize to ImageNet?](https://mlsanity.com/paper/1902.10811/) (2019)
- [EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classification](https://mlsanity.com/paper/1709.00029/) (2017)
- [Conditional Prompt Learning for Vision-Language Models](https://mlsanity.com/paper/2203.05557/) (2022)
- [Natural Adversarial Examples](https://mlsanity.com/paper/1907.07174/) (2019)
- [In Search of Lost Domain Generalization](https://mlsanity.com/paper/2007.01434/) (2020)
- [Learning Robust Global Representations by Penalizing Local Predictive Power](https://mlsanity.com/paper/1905.13549/) (2019)
- [How Can We Know What Language Models Know?](https://mlsanity.com/paper/1911.12543/) (2019)
- [Causal inference using invariant prediction: identification and confidence intervals](https://mlsanity.com/paper/1501.01332/) (2015)
- [DenseCLIP: Language-Guided Dense Prediction with Context-Aware Prompting](https://mlsanity.com/paper/2112.01518/) (2021)
- [Test-Time Prompt Tuning for Zero-Shot Generalization in Vision-Language Models](https://mlsanity.com/paper/2209.07511/) (2022)
- [Prompt-aligned Gradient for Prompt Tuning](https://mlsanity.com/paper/2205.14865/) (2022)
- [Learning to Prompt for Open-Vocabulary Object Detection with Vision-Language Model](https://mlsanity.com/paper/2203.14940/) (2022)

## Built on by (most cited first, 30 of 51)

- [Consistency-guided Prompt Learning for Vision-Language Models](https://mlsanity.com/paper/2306.01195/) (2023)
- [Learning to Prompt with Text Only Supervision for Vision-Language Models](https://mlsanity.com/paper/2401.02418/) (2024)
- [CLAP4CLIP: Continual Learning with Probabilistic Finetuning for Vision-Language Models](https://mlsanity.com/paper/2403.19137/) (2024)
- [Prompting Language-Informed Distribution for Compositional Zero-Shot Learning](https://mlsanity.com/paper/2305.14428/) (2023)
- [Any-Shift Prompting for Generalization over Distributions](https://mlsanity.com/paper/2402.10099/) (2024)
- [Soft Prompt Generation for Domain Generalization](https://mlsanity.com/paper/2404.19286/) (2024)
- [Adapting to Distribution Shift by Visual Domain Prompt Generation](https://mlsanity.com/paper/2405.02797/) (2024)
- [Advances in Multimodal Adaptation and Generalization: From Traditional Approaches to Foundation Models](https://mlsanity.com/paper/2501.18592/) (2025)
- [PromptFusion: Decoupling Stability and Plasticity for Continual Learning](https://mlsanity.com/paper/2303.07223/) (2023)
- [IPO: Interpretable Prompt Optimization for Vision-Language Models](https://mlsanity.com/paper/2410.15397/) (2024)
- [Make Prompts Adaptable: Bayesian Modeling for Vision-Language Prompt Learning with Data-Dependent Prior](https://mlsanity.com/paper/2401.06799/) (2024)
- [Active Generation for Image Classification](https://mlsanity.com/paper/2403.06517/) (2024)
- [Aligning Medical Images with General Knowledge from Large Language Models](https://mlsanity.com/paper/2409.00341/) (2024)
- [A Closer Look at Conditional Prompt Tuning for Vision-Language Models](https://mlsanity.com/paper/2506.23856/) (2025)
- [Tree of Attributes Prompt Learning for Vision-Language Models](https://mlsanity.com/paper/2410.11201/) (2024)
- [Descriptor and Word Soups: Overcoming the Parameter Efficiency Accuracy Tradeoff for Out-of-Distribution Few-shot Learning](https://mlsanity.com/paper/2311.13612/) (2023)
- [Referring Expression Comprehension for Small Objects](https://mlsanity.com/paper/2510.03701/) (2025)
- [Style-Pro: Style-Guided Prompt Learning for Generalizable Vision-Language Models](https://mlsanity.com/paper/2411.16018/) (2024)
- [Medical Knowledge Intervention Prompt Tuning for Medical Image Classification](https://mlsanity.com/paper/2511.12639/) (2025)
- [Bayesian Principles Improve Prompt Learning In Vision-Language Models](https://mlsanity.com/paper/2504.14123/) (2025)
- [DiSa: Directional Saliency-Aware Prompt Learning for Generalizable Vision-Language Models](https://mlsanity.com/paper/2505.19373/) (2025)
- [Knowledge Regularized Negative Feature Tuning of Vision-Language Models for Out-of-Distribution Detection](https://mlsanity.com/paper/2507.19847/) (2025)
- [Crane: Context-Guided Prompt Learning and Attention Refinement for Zero-Shot Anomaly Detection](https://mlsanity.com/paper/2504.11055/) (2025)
- [Toward a Holistic Approach to Continual Model Merging](https://mlsanity.com/paper/2509.23592/) (2025)
- [VaMP: Variational Multi-Modal Prompt Learning for Vision-Language Models](https://mlsanity.com/paper/2511.22664/) (2025)
- [Embracing Diversity: Interpretable Zero-shot classification beyond one vector per class](https://mlsanity.com/paper/2404.16717/) (2024)
- [CVPT: Cross Visual Prompt Tuning](https://mlsanity.com/paper/2408.14961/) (2024)
- [Prompt Diffusion Robustifies Any-Modality Prompt Learning](https://mlsanity.com/paper/2410.20164/) (2024)
- [Modeling Variants of Prompts for Vision-Language Models](https://mlsanity.com/paper/2503.08229/) (2025)
- [Craft: Cross-modal Aligned Features Improve Robustness of Prompt Tuning](https://mlsanity.com/paper/2407.15894/) (2024)

## Abstract (arXiv)

Foundational image-language models have generated considerable interest due to their efficient adaptation to downstream tasks by prompt learning. Prompt learning treats part of the language model input as trainable while freezing the rest, and optimizes an Empirical Risk Minimization objective. However, Empirical Risk Minimization is known to suffer from distributional shifts which hurt generalizability to prompts unseen during training. By leveraging the regularization ability of Bayesian methods, we frame prompt learning from the Bayesian perspective and formulate it as a variational inference problem. Our approach regularizes the prompt space, reduces overfitting to the seen prompts and improves the prompt generalization on unseen prompts. Our framework is implemented by modeling the input prompt space in a probabilistic manner, as an a priori distribution which makes our proposal compatible with prompt learning approaches that are unconditional or conditional on the image. We demonstrate empirically on 15 benchmarks that Bayesian prompt learning provides an appropriate coverage of the prompt space, prevents learning spurious features, and exploits transferable invariant features. This results in better generalization of unseen prompts, even across different datasets and domains. Code available at: https://github.com/saic-fi/Bayesian-Prompt-Learning