mlsanityData to 14 Oct 2026

Probabilistic Embeddings for Frozen Vision-Language Models: Uncertainty Quantification with Gaussian Process Latent Variable Models

arXiv 2505.05163 ↗First posted 8 May 2025cs.CV cs.LGcited by 9

On the map: Multimodal LLMs / vision-language models → Hallucination mitigation in LVLMs

Abstract (arXiv)

Vision-Language Models (VLMs) learn joint representations by mapping images and text into a shared latent space. However, recent research highlights that deterministic embeddings from standard VLMs often struggle to capture the uncertainties arising from the ambiguities in visual and textual descriptions and the multiple possible correspondences between images and texts. Existing approaches tackle this by learning probabilistic embeddings during VLM training, which demands large datasets and does not leverage the powerful representations already learned by large-scale VLMs like CLIP. In this paper, we propose GroVE, a post-hoc approach to obtaining probabilistic embeddings from frozen VLMs. GroVE builds on Gaussian Process Latent Variable Model (GPLVM) to learn a shared low-dimensional latent space where image and text inputs are mapped to a unified representation, optimized through single-modal embedding reconstruction and cross-modal alignment objectives. Once trained, the Gaussian Process model generates uncertainty-aware probabilistic embeddings. Evaluation shows that GroVE achieves state-of-the-art uncertainty calibration across multiple downstream tasks, including cross-modal retrieval, visual question answering, and active learning.

Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Placed in Hallucination mitigation in LVLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 43 of its 62 references that are in the map. First posted on arXiv 8 May 2025.

Builds on

43 of 62 references.

Closest ancestors · 1

Same direction, newest first

  1. 2022MAP: Multimodal Uncertainty-Aware Vision-Language Pre-training Model11 Oct

Other references · 42

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  4. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  5. 2017On Calibration of Modern Neural Networkscited by 4,960
  6. 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,448
  7. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  8. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
  9. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,222
  10. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,553
  11. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,462
  12. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,296
30 more
  1. 2023Vision-Language Models for Vision Tasks: A Surveycited by 839
  2. 2020A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challengescited by 819
  3. 2018Evidential Deep Learning to Quantify Classification Uncertaintycited by 732
  4. 2021FLAVA: A Foundational Language And Vision Alignment Modelcited by 649
  5. 2018GPyTorch: Blackbox Matrix-Matrix Gaussian Process Inference with GPU Accelerationcited by 609
  6. 2020Simple and Principled Uncertainty Estimation with Deterministic Deep Learning via Distance Awarenesscited by 389
  7. 2018Neural Processescited by 311
  8. 2018Aleatoric uncertainty estimation with test-time augmentation for medical image segmentation with convolutional neural networkscited by 280
  9. 2014Scalable Variational Gaussian Process Classificationcited by 228
  10. 2021Deep Deterministic Uncertainty: A Simple Baselinecited by 200
  11. 2016Learning Deep Representations of Fine-grained Visual Descriptionscited by 182
  12. 2019Probabilistic Face Embeddingscited by 173
  13. 2023CORA: Adapting CLIP for Open-Vocabulary Detection with Region Prompting and Anchor Pre-Matchingcited by 133
  14. 2015Zero-Shot Learning via Semantic Similarity Embeddingcited by 74
  15. 2023CLIP-Guided Vision-Language Pre-training for Question Answering in 3D Scenescited by 55
  16. 2018Hierarchy-based Image Embeddings for Semantic Image Retrievalcited by 41
  17. 2023Improved Probabilistic Image-Text Representationscited by 40
  18. 2020Confidence Estimation via Auxiliary Modelscited by 34
  19. 2024Online Zero-Shot Classification with CLIPcited by 22
  20. 2023GridCLIP: One-Stage Object Detection by Grid-Level CLIP Representation Learningcited by 15
  21. 2021SLURP: Side Learning Uncertainty for Regression Problemscited by 12
  22. 2024CLIPVQA:Video Quality Assessment via CLIPcited by 11
  23. 2022Uncertainty Estimation for Multi-view Data: The Power of Seeing the Whole Picturecited by 10
  24. 2023Gaussian Latent Representations for Uncertainty Estimation using Mahalanobis Distance in Deep Classifierscited by 10
  25. 2023Out-of-Distribution Detection for Monocular Depth Estimationcited by 7
  26. 2021Tackling Inter-Class Similarity and Intra-Class Variance for Microscopic Image-based Classificationcited by 6
  27. 2022Generalised Gaussian Process Latent Variable Models (GPLVM) with Stochastic Variational Inferencecited by 5
  28. 2020Towards Zero-Shot Learning with Fewer Seen Class Examplescited by 4
  29. 2023Integrating Visual and Semantic Similarity Using Hierarchies for Image Retrievalcited by 3
  30. 2017Learning Structured Semantic Embeddings for Visual Recognitioncited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 7.

9 papers.

  1. 2025Distance-informed Neural Processescited by 3
  2. 2026VLM-UQBench: A Benchmark for Modality-Specific and Cross-Modality Uncertainties in Vision Language Modelscited by 3
  3. 2026Remember with Confidence: Uncertainty Quantification for Spatio-temporal Memory with Probabilistic Guaranteescited by 3
  4. 2026Epistemic Uncertainty Quantification for Pre-trained VLMs via Riemannian Flow Matchingcited by 2
  5. 2026FUSE: Quantifying Uncertainty in Vision-Language Models by Bayesian Fusing Epistemic and Aleatoric Uncertaintycited by 1
  6. 2025Uncertainty-driven Embedding Convolutioncited by 0
  7. 2026Logit-Level Uncertainty Quantification in Vision-Language Models for Histopathology Image Analysiscited by 0
  8. 2026GeoFlowVLM: Geometry-Aware Joint Uncertainty for Frozen Vision-Language Embeddingcited by 0
  9. 2026General Incomplete Multimodal Learning via Dynamic Quality Perceptioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023ProbVLM: Probabilistic Adapter for Frozen Vision-Language Models
  2. 2025Exploiting the Asymmetric Uncertainty Structure of Pre-trained VLMs on the Unit Hypersphere
  3. 2024Probabilistic Language-Image Pre-Training
  4. 2025Intra-Class Probabilistic Embeddings for Uncertainty Estimation in Vision-Language Models
  5. 2024Post-hoc Probabilistic Vision-Language Models
  6. 2025ViLU: Learning Vision-Language Uncertainties for Failure Prediction
  7. 2025FrEVL: Leveraging Frozen Pretrained Embeddings for Efficient Vision-Language Understanding
  8. 2025Interpreting the linear structure of vision-language model embedding spaces