mlsanityData to 7 Oct 2026

Post-hoc Probabilistic Vision-Language Models

arXiv 2412.06014 ↗First posted 8 Dec 2024cs.CV cs.LGcited by 13

On the map: Multimodal LLMs / vision-language models → Hallucination mitigation in LVLMs

Abstract (arXiv)

Vision-language models (VLMs), such as CLIP and SigLIP, have found remarkable success in classification, retrieval, and generative tasks. For this, VLMs deterministically map images and text descriptions to a joint latent space in which their similarity is assessed using the cosine similarity. However, a deterministic mapping of inputs fails to capture uncertainties over concepts arising from domain shifts when used in downstream tasks. In this work, we propose post-hoc uncertainty estimation in VLMs that does not require additional training. Our method leverages a Bayesian posterior approximation over the last layers in VLMs and analytically quantifies uncertainties over cosine similarities. We demonstrate its effectiveness for uncertainty quantification and support set selection in active learning. Compared to baselines, we obtain improved and well-calibrated predictive uncertainties, interpretable uncertainty estimates, and sample-efficient active learning. Our results show promise for safety-critical applications of large-scale models.

Key facts. Cited by 13 papers in the map (data through 7 Oct 2026). Placed in Hallucination mitigation in LVLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 66 of its 90 references that are in the map. First posted on arXiv 8 Dec 2024.

Builds on

66 of 90 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024Probabilistic Language-Image Pre-Training24 Oct
  2. 2023ProbVLM: Probabilistic Adapter for Frozen Vision-Language Models1 Jul
  3. 2022MAP: Multimodal Uncertainty-Aware Vision-Language Pre-training Model11 Oct

Other references · 63

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  3. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  4. 2017On Calibration of Modern Neural Networkscited by 4,860
  5. 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,416
  6. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,290
  7. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  8. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,888
  9. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  10. 2017What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?cited by 2,335
  11. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,842
  12. 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,488
38 more
  1. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
  2. 2017Deep Hashing Network for Unsupervised Domain Adaptationcited by 1,181
  3. 2019Learning Robust Global Representations by Penalizing Local Predictive Powercited by 1,010
  4. 2023Vision-Language Models for Vision Tasks: A Surveycited by 834
  5. 2017Deep Bayesian Active Learning with Image Datacited by 812
  6. 2015Optimizing Neural Networks with Kronecker-factored Approximate Curvaturecited by 776
  7. 2023DataComp: In search of the next generation of multimodal datasetscited by 635
  8. 2019Deep Batch Active Learning by Diverse, Uncertain Gradient Lower Boundscited by 602
  9. 2020A Survey of Deep Active Learningcited by 547
  10. 2019Measuring Calibration in Deep Learningcited by 443
  11. 2021Laplace Redux -- Effortless Bayesian Deep Learningcited by 359
  12. 2019Variational Adversarial Active Learningcited by 284
  13. 2021Exploring the Limits of Out-of-Distribution Detectioncited by 283
  14. 2022Prompt Distribution Learningcited by 235
  15. 2021RedCaps: web-curated image-text data created by the people, for the peoplecited by 175
  16. 2024An Introduction to Vision-Language Modelingcited by 175
  17. 2023Pic2Word: Mapping Pictures to Words for Zero-shot Composed Image Retrievalcited by 155
  18. 2021Active Learning by Acquiring Contrastive Examplescited by 154
  19. 2021Probabilistic Embeddings for Cross-Modal Retrievalcited by 150
  20. 2020Contextual Diversity for Active Learningcited by 139
  21. 2020Better Aggregation in Test-Time Augmentationcited by 121
  22. 2023Bayesian Low-rank Adaptation for Large Language Modelscited by 112
  23. 2021Scalable Marginal Likelihood Estimation for Model Selection in Deep Learningcited by 105
  24. 2023Vision-by-Language for Training-Free Compositional Image Retrievalcited by 102
  25. 2020Bayesian Deep Learning via Subnetwork Inferencecited by 85
  26. 2024C-TPT: Calibrated Test-Time Prompt Tuning for Vision-Language Models via Text Feature Dispersioncited by 82
  27. 2024Variational Learning is Effective for Large Deep Networkscited by 62
  28. 2023Prediction-Oriented Bayesian Active Learningcited by 60
  29. 2024Position: Bayesian Deep Learning is Needed in the Age of Large-Scale AIcited by 60
  30. 2022Uncertainty-guided Source-free Domain Adaptationcited by 46
  31. 2024A Closer Look at the Robustness of Contrastive Language-Image Pre-Training (CLIP)cited by 44
  32. 2024Efficiently Learning at Test-Time: Active Fine-Tuning of LLMscited by 42
  33. 2023Improved Probabilistic Image-Text Representationscited by 40
  34. 2023Active Finetuning: Exploiting Annotation Budget in the Pretraining-Finetuning Paradigmcited by 36
  35. 2024Frustratingly Easy Test-Time Adaptation of Vision-Language Modelscited by 36
  36. 2023What Can We Learn From The Selective Prediction And Uncertainty Estimation Performance Of 523 Imagenet Classifierscited by 35
  37. 2022Post-hoc Uncertainty Learning using a Dirichlet Meta-Modelcited by 31
  38. 2022Probabilistic Compositional Embeddings for Multimodal Image Retrievalcited by 29

50 most cited of 63 All 63 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 8; 2026: 5.

13 papers.

Sort:

By area:

  1. 2026InfoNCE Induces Gaussian Distributioncited by 14
  2. 2025ViLU: Learning Vision-Language Uncertainties for Failure Predictioncited by 9
  3. 2025Ensembling Pruned Attention Heads For Uncertainty-Aware Efficient Transformerscited by 5
  4. 2025Exploiting the Asymmetric Uncertainty Structure of Pre-trained VLMs on the Unit Hyperspherecited by 4
  5. 2025Multiplicity is an Inevitable and Inherent Challenge in Multimodal Learningcited by 4
  6. 2025Improving Semantic Uncertainty Quantification in LVLMs with Semantic Gaussian Processescited by 3
  7. 2026EAGOR: Embodied Reasoning in Omni-directioncited by 3
  8. 2025Intra-Class Probabilistic Embeddings for Uncertainty Estimation in Vision-Language Modelscited by 2
  9. 2026Epistemic Uncertainty Quantification for Pre-trained VLMs via Riemannian Flow Matchingcited by 2
  10. 2025PARIC: Probabilistic Attention Regularization for Language Guided Image Classification from Pre-trained Vison Language Modelscited by 1
  11. 2025Uncertainty-driven Embedding Convolutioncited by 0
  12. 2026GeoFlowVLM: Geometry-Aware Joint Uncertainty for Frozen Vision-Language Embeddingcited by 0
1 more
  1. 2026Leveraging Visual Signals for Robust Token-Level Uncertainty in Vision-Language Generationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Uncertainty-Aware Evaluation for Vision-Language Models
  2. 2025Probabilistic Embeddings for Frozen Vision-Language Models: Uncertainty Quantification with Gaussian Process Latent Variable Models
  3. 2026FUSE: Quantifying Uncertainty in Vision-Language Models by Bayesian Fusing Epistemic and Aleatoric Uncertainty
  4. 2026Dynamic Distribution-Aware Uncertainty Tracking in Vision-Language Representation Learning
  5. 2025Optimizing Active Learning in Vision-Language Models via Parameter-Efficient Uncertainty Calibration
  6. 2026VLM-UQBench: A Benchmark for Modality-Specific and Cross-Modality Uncertainties in Vision Language Models
  7. 2025The Art of Saying "Maybe": A Conformal Lens for Uncertainty Benchmarking in VLMs
  8. 2025Test-Time Consistency in Vision Language Models