mlsanityData to 7 Oct 2026

Intra-Class Probabilistic Embeddings for Uncertainty Estimation in Vision-Language Models

arXiv 2511.22019 ↗First posted 27 Nov 2025cs.CVcited by 2

On the map: Multimodal LLMs / vision-language models → Hallucination mitigation in LVLMs

Abstract (arXiv)

Vision-language models (VLMs), such as CLIP, have gained popularity for their strong open vocabulary classification performance, but they are prone to assigning high confidence scores to misclassifications, limiting their reliability in safety-critical applications. We introduce a training-free, post-hoc uncertainty estimation method for contrastive VLMs that can be used to detect erroneous predictions. The key to our approach is to measure visual feature consistency within a class, using feature projection combined with multivariate Gaussians to create class-specific probabilistic embeddings. Our method is VLM-agnostic, requires no fine-tuning, demonstrates robustness to distribution shift, and works effectively with as few as 10 training images per class. Extensive experiments on ImageNet, Flowers102, Food101, EuroSAT and DTD show state-of-the-art error detection performance, significantly outperforming both deterministic and probabilistic VLM baselines. Code is available at https://github.com/zhenxianglin/ICPE.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Hallucination mitigation in LVLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 45 of its 58 references that are in the map. First posted on arXiv 27 Nov 2025.

Builds on

45 of 58 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024Post-hoc Probabilistic Vision-Language Models8 Dec
  2. 2024Probabilistic Language-Image Pre-Training24 Oct
  3. 2022MAP: Multimodal Uncertainty-Aware Vision-Language Pre-training Model11 Oct

Other references · 42

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2017On Calibration of Modern Neural Networkscited by 4,860
  3. 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,416
  4. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,888
  5. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  6. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  7. 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,857
  8. 2016A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networkscited by 2,362
  9. 2017What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?cited by 2,335
  10. 2015Variational Inference with Normalizing Flowscited by 2,161
  11. 2013Describing Textures in the Wildcited by 2,110
  12. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,507
30 more
  1. 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,488
  2. 2019Natural Adversarial Examplescited by 1,401
  3. 2017Enhancing The Reliability of Out-of-distribution Image Detection in Neural Networkscited by 1,325
  4. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,265
  5. 2023ImageBind: One Embedding Space To Bind Them Allcited by 1,162
  6. 2018Evidential Deep Learning to Quantify Classification Uncertaintycited by 728
  7. 2018Predictive Uncertainty Estimation via Prior Networkscited by 555
  8. 2022Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learningcited by 553
  9. 2020BatchEnsemble: An Alternative Approach to Efficient Ensemble and Lifelong Learningcited by 368
  10. 2023LanguageBind: Extending Video-Language Pretraining to N-modality by Language-based Semantic Alignmentcited by 366
  11. 2022Visual Language Maps for Robot Navigationcited by 362
  12. 2022CoWs on Pasture: Baselines and Benchmarks for Language-Driven Zero-Shot Object Navigationcited by 194
  13. 2025RoboBrain: A Unified Brain Model for Robotic Manipulation from Abstract to Concretecited by 134
  14. 2023CLIPN for Zero-Shot OOD Detection: Teaching CLIP to Say Nocited by 130
  15. 2017Bayesian Compression for Deep Learningcited by 116
  16. 2023Nearest Neighbor Guidance for Out-of-Distribution Detectioncited by 64
  17. 2018Direct Uncertainty Prediction for Medical Second Opinionscited by 61
  18. 2024A Closer Look at the Robustness of Contrastive Language-Image Pre-Training (CLIP)cited by 44
  19. 2024The Double-Ellipsoid Geometry of CLIPcited by 42
  20. 2023Improved Probabilistic Image-Text Representationscited by 40
  21. 2019Deep Sub-Ensembles for Fast Uncertainty Estimation in Image Classificationcited by 36
  22. 2024Frustratingly Easy Test-Time Adaptation of Vision-Language Modelscited by 36
  23. 2024MediCLIP: Adapting CLIP for Few-shot Medical Image Anomaly Detectioncited by 35
  24. 2024An Empirical Study Into What Matters for Calibrating Vision-Language Modelscited by 20
  25. 2023Enabling Calibration In The Zero-Shot Inference of Large Vision-Language Modelscited by 17
  26. 2024Open-Set Recognition in the Age of Vision-Language Modelscited by 13
  27. 2023CLIP-Motion: Learning Reward Functions for Robotic Actions Using Consecutive Observationscited by 8
  28. 2024EndoOOD: Uncertainty-aware Out-of-distribution Detection in Capsule Endoscopy Diagnosiscited by 8
  29. 2020Multi-Loss Sub-Ensembles for Accurate Classification with Uncertainty Estimationcited by 5
  30. 2025To Trust Or Not To Trust Your Vision-Language Model's Predictioncited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026VISTA: Auditing Semantic Divergence in Vision-Language Modelscited by 0
  2. 2026General Incomplete Multimodal Learning via Dynamic Quality Perceptioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Exploiting the Asymmetric Uncertainty Structure of Pre-trained VLMs on the Unit Hypersphere
  2. 2025Probabilistic Embeddings for Frozen Vision-Language Models: Uncertainty Quantification with Gaussian Process Latent Variable Models
  3. 2023ProbVLM: Probabilistic Adapter for Frozen Vision-Language Models
  4. 2025ViLU: Learning Vision-Language Uncertainties for Failure Prediction
  5. 2024Uncertainty-Aware Evaluation for Vision-Language Models
  6. 2026FUSE: Quantifying Uncertainty in Vision-Language Models by Bayesian Fusing Epistemic and Aleatoric Uncertainty
  7. 2025Representation Calibration and Uncertainty Guidance for Class-Incremental Learning based on Vision Language Model
  8. 2026Dynamic Distribution-Aware Uncertainty Tracking in Vision-Language Representation Learning