mlsanityData to 7 Oct 2026

ProbVLM: Probabilistic Adapter for Frozen Vision-Language Models

arXiv 2307.00398 ↗First posted 1 Jul 2023cs.CV cs.AI cs.LGcited by 7

On the map: Multimodal LLMs / vision-language models → Hallucination mitigation in LVLMs

Abstract (arXiv)

Large-scale vision-language models (VLMs) like CLIP successfully find correspondences between images and text. Through the standard deterministic mapping process, an image or a text sample is mapped to a single vector in the embedding space. This is problematic: as multiple samples (images or text) can abstract the same concept in the physical world, deterministic embeddings do not reflect the inherent ambiguity in the embedding space. We propose ProbVLM, a probabilistic adapter that estimates probability distributions for the embeddings of pre-trained VLMs via inter/intra-modal alignment in a post-hoc manner without needing large-scale datasets or computing. On four challenging datasets, i.e., COCO, Flickr, CUB, and Oxford-flowers, we estimate the multi-modal embedding uncertainties for two VLMs, i.e., CLIP and BLIP, quantify the calibration of embedding uncertainties in retrieval tasks and show that ProbVLM outperforms other methods. Furthermore, we propose active learning and model selection as two real-world downstream tasks for VLMs and show that the estimated uncertainty aids both tasks. Lastly, we present a novel technique for visualizing the embedding distributions using a large-scale pre-trained latent diffusion model. Code is available at https://github.com/ExplainableML/ProbVLM.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Placed in Hallucination mitigation in LVLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 73 of its 107 references that are in the map. First posted on arXiv 1 Jul 2023.

Builds on

73 of 107 references.

References · 73

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  4. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  5. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  6. 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,416
  7. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  8. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  9. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  10. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,888
  11. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  12. 2017What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?cited by 2,335
38 more
  1. 2021Learning to Prompt for Vision-Language Modelscited by 2,269
  2. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  3. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
  4. 2022Conditional Prompt Learning for Vision-Language Modelscited by 1,427
  5. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,366
  6. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,352
  7. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,265
  8. 2018Averaging Weights Leads to Wider Optima and Better Generalizationcited by 1,194
  9. 2022Editing Models with Task Arithmeticcited by 1,076
  10. 2021CLIP-Adapter: Better Vision-Language Models with Feature Adapterscited by 1,021
  11. 2021Robust fine-tuning of zero-shot modelscited by 785
  12. 2021FLAVA: A Foundational Language And Vision Alignment Modelcited by 646
  13. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervisioncited by 645
  14. 2021FILIP: Fine-grained Interactive Language-Image Pre-Trainingcited by 626
  15. 2021SLIP: Self-supervision meets Language-Image Pre-trainingcited by 434
  16. 2019BatchBALD: Efficient and Diverse Batch Acquisition for Deep Bayesian Active Learningcited by 428
  17. 2019Selection via Proxy: Efficient Data Selection for Deep Learningcited by 337
  18. 2018Aleatoric uncertainty estimation with test-time augmentation for medical image segmentation with convolutional neural networkscited by 278
  19. 2022Delving into Out-of-Distribution Detection with Vision-Language Representationscited by 215
  20. 2022Patching open-vocabulary models by interpolating weightscited by 187
  21. 2016Learning Deep Representations of Fine-grained Visual Descriptionscited by 182
  22. 2019Probabilistic Face Embeddingscited by 171
  23. 2021Probabilistic Embeddings for Cross-Modal Retrievalcited by 150
  24. 2022Leveraging Unlabeled Data to Predict Out-of-Distribution Performancecited by 141
  25. 2021Predicting with Confidence on Unseen Distributionscited by 116
  26. 2021Estimating and Exploiting the Aleatoric Uncertainty in Surface Normal Estimationcited by 116
  27. 2020Are Labels Always Necessary for Classifier Accuracy Evaluation?cited by 104
  28. 2022Plex: Towards Reliability using Pretrained Large Model Extensionscited by 102
  29. 2022Reliable Visual Question Answering: Abstain Rather Than Answer Incorrectlycited by 86
  30. 2019Polysemous Visual-Semantic Embedding for Cross-Modal Retrievalcited by 83
  31. 2018Lightweight Probabilistic Deep Networkscited by 77
  32. 2021What Does Rotation Prediction Tell Us about Classifier Accuracy under Varying Testing Environments?cited by 63
  33. 2016A Comprehensive Survey on Cross-modal Retrievalcited by 62
  34. 2022A CLIP-Hitchhiker's Guide to Long Video Retrievalcited by 57
  35. 2020Estimating Generalization under Distribution Shifts via Domain-Invariant Representationscited by 56
  36. 2021Towards Fewer Annotations: Active Learning via Region Impurity and Prediction Uncertainty for Domain Adaptive Semantic Segmentationcited by 56
  37. 2021Mandoline: Model Evaluation under Distribution Shiftcited by 55
  38. 2023UATVR: Uncertainty-Adaptive Text-Video Retrievalcited by 55

50 most cited of 73 All 73 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 1; 2025: 4; 2026: 1.

7 papers.

  1. 2024Post-hoc Probabilistic Vision-Language Modelscited by 13
  2. 2025Exploiting the Asymmetric Uncertainty Structure of Pre-trained VLMs on the Unit Hyperspherecited by 4
  3. 2025Multiplicity is an Inevitable and Inherent Challenge in Multimodal Learningcited by 4
  4. 2025Entropy Heat-Mapping: Localizing GPT-Based OCR Errors with Sliding-Window Shannon Analysiscited by 2
  5. 2023HypUC: Hyperfine Uncertainty Calibration with Gradient-boosted Corrections for Reliable Regression on Imbalanced Electrocardiogramscited by 1
  6. 2025PVeRA: Probabilistic Vector-Based Random Matrix Adaptationcited by 0
  7. 2026Emergent Communication between Heterogeneous Visual Agents through Decentralized Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Probabilistic Embeddings for Frozen Vision-Language Models: Uncertainty Quantification with Gaussian Process Latent Variable Models
  2. 2024Probabilistic Language-Image Pre-Training
  3. 2025Intra-Class Probabilistic Embeddings for Uncertainty Estimation in Vision-Language Models
  4. 2023Revisiting the Role of Language Priors in Vision-Language Models
  5. 2023MLLMs-Augmented Visual-Language Representation Learning
  6. 2026From Inference to Adaptation: A Unified Optimal Transport View of Vision Language Model
  7. 2024Imperfect Vision Encoders: Efficient and Robust Tuning for Vision-Language Models
  8. 202620/20 Vision Language Models: A Prescription for Better VLMs through Data Curation Alone