mlsanityData to 14 Oct 2026

Probabilistic Language-Image Pre-Training

arXiv 2410.18857 ↗First posted 24 Oct 2024cs.CV cs.LGcited by 16

On the map: Multimodal LLMs / vision-language models → Hallucination mitigation in LVLMs

Abstract (arXiv)

Vision-language models (VLMs) embed aligned image-text pairs into a joint space but often rely on deterministic embeddings, assuming a one-to-one correspondence between images and texts. This oversimplifies real-world relationships, which are inherently many-to-many, with multiple captions describing a single image and vice versa. We introduce Probabilistic Language-Image Pre-training (ProLIP), the first probabilistic VLM pre-trained on a billion-scale image-text dataset using only probabilistic objectives, achieving a strong zero-shot capability (e.g., 74.6% ImageNet zero-shot accuracy with ViT-B/16). ProLIP efficiently estimates uncertainty by an "uncertainty token" without extra parameters. We also introduce a novel inclusion loss that enforces distributional inclusion relationships between image-text pairs and between original and masked inputs. Experiments demonstrate that, by leveraging uncertainty estimates, ProLIP benefits downstream tasks and aligns with intuitive notions of uncertainty, e.g., shorter texts being more uncertain and more general inputs including specific ones. Utilizing text uncertainties, we further improve ImageNet accuracy from 74.6% to 75.8% (under a few-shot setting), supporting the practical advantages of our probabilistic approach. The code is available at https://github.com/naver-ai/prolip

Key facts. Cited by 16 papers in the map (data through 14 Oct 2026). Placed in Hallucination mitigation in LVLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 54 of its 78 references that are in the map. First posted on arXiv 24 Oct 2024.

Builds on

54 of 78 references.

Closest ancestors · 1

Same direction, newest first

  1. 2022MAP: Multimodal Uncertainty-Aware Vision-Language Pre-training Model11 Oct

Other references · 53

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  6. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
  7. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  8. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,415
  9. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,183
  10. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
  11. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  12. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
38 more
  1. 2019CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Featurescited by 2,884
  2. 2013Describing Textures in the Wildcited by 2,125
  3. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,850
  4. 2013Fine-Grained Visual Classification of Aircraftcited by 1,726
  5. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,515
  6. 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,496
  7. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,462
  8. 2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,457
  9. 2019Natural Adversarial Examplescited by 1,408
  10. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,216
  11. 2022Reproducible scaling laws for contrastive language-image learningcited by 1,173
  12. 2019Learning Robust Global Representations by Penalizing Local Predictive Powercited by 1,013
  13. 2017Remote Sensing Image Scene Classification: Benchmark and State of the Artcited by 797
  14. 2023DataComp: In search of the next generation of multimodal datasetscited by 644
  15. 2019A Large-scale Study of Representation Learning with the Visual Task Adaptation Benchmarkcited by 425
  16. 2018Rotation Equivariant CNNs for Digital Pathologycited by 422
  17. 2017Functional Map of the Worldcited by 338
  18. 2020Are we done with ImageNet?cited by 333
  19. 2022Scaling Language-Image Pre-training via Maskingcited by 317
  20. 2024Long-CLIP: Unlocking the Long-Text Capability of CLIPcited by 300
  21. 2023Data Filtering Networkscited by 250
  22. 2021RedCaps: web-curated image-text data created by the people, for the peoplecited by 180
  23. 2019Probabilistic Face Embeddingscited by 173
  24. 2021Probabilistic Embeddings for Cross-Modal Retrievalcited by 151
  25. 2023Hyperbolic Image-Text Representationscited by 136
  26. 2020Data Uncertainty Learning in Face Recognitioncited by 134
  27. 2021Re-labeling ImageNet: from Single to Multi-Labels, from Global to Localized Labelscited by 118
  28. 2021The iWildCam 2021 Competition Datasetcited by 77
  29. 2018Modeling Uncertainty with Hedged Instance Embeddingcited by 52
  30. 2023GeoDE: a Geographically Diverse Evaluation Dataset for Object Recognitioncited by 50
  31. 2019View-Invariant Probabilistic Embedding for Human Posecited by 47
  32. 2023Improved Probabilistic Image-Text Representationscited by 40
  33. 2022ECCV Caption: Correcting False Negatives by Collecting Machine-and-Human-verified Image-Caption Associations for MS-COCOcited by 37
  34. 2022Probabilistic Representations for Video Contrastive Learningcited by 34
  35. 2022Probabilistic Compositional Embeddings for Multimodal Image Retrievalcited by 29
  36. 2023Probabilistic Contrastive Learning Recovers the Correct Aleatoric Uncertainty of Ambiguous Inputscited by 29
  37. 2022WinoGAViL: Gamified Association Benchmark to Challenge Vision-and-Language Modelscited by 20
  38. 2024Emergent Visual-Semantic Hierarchies in Image-Text Representationscited by 19

50 most cited of 53 All 53 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 11; 2026: 4.

16 papers.

Sort:

By area:

  1. 2024Post-hoc Probabilistic Vision-Language Modelscited by 13
  2. 2025Explaining Similarity in Vision-Language Encoders with Weighted Banzhaf Interactionscited by 6
  3. 2025Visual Instruction Bottleneck Tuningcited by 5
  4. 2025Multiplicity is an Inevitable and Inherent Challenge in Multimodal Learningcited by 4
  5. 2025Global and Local Entailment Learning for Natural World Imagerycited by 3
  6. 2025Intra-Class Probabilistic Embeddings for Uncertainty Estimation in Vision-Language Modelscited by 2
  7. 2026Epistemic Uncertainty Quantification for Pre-trained VLMs via Riemannian Flow Matchingcited by 2
  8. 2025LongProLIP: A Probabilistic Vision-Language Model with Long Context Textcited by 1
  9. 2025DALIP: Distribution Alignment-based Language-Image Pre-Training for Domain-Specific Datacited by 1
  10. 2025ProM3E: Probabilistic Masked MultiModal Embedding Model for Ecologycited by 1
  11. 2026ARGENT: Adaptive Hierarchical Image-Text Representationscited by 1
  12. 2026CoMet: Context and Multiplicity Decomposition for Multimodal Uncertainty Estimationcited by 1
4 more
  1. 2025Descriptive Image-Text Matching with Graded Contextual Similaritycited by 0
  2. 2025Emergence of Text Readability in Vision Language Modelscited by 0
  3. 2025Uncertainty-driven Embedding Convolutioncited by 0
  4. 2026GeoFlowVLM: Geometry-Aware Joint Uncertainty for Frozen Vision-Language Embeddingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023ProbVLM: Probabilistic Adapter for Frozen Vision-Language Models
  2. 2025Exploiting the Asymmetric Uncertainty Structure of Pre-trained VLMs on the Unit Hypersphere
  3. 2025Probabilistic Embeddings for Frozen Vision-Language Models: Uncertainty Quantification with Gaussian Process Latent Variable Models
  4. 2023Revisiting the Role of Language Priors in Vision-Language Models
  5. 2025Trust the Model: Compact VLMs as In-Context Judges for Image-Text Data Quality
  6. 202620/20 Vision Language Models: A Prescription for Better VLMs through Data Curation Alone
  7. 2024NEVLP: Noise-Robust Framework for Efficient Vision-Language Pre-training
  8. 2024DreamLIP: Language-Image Pre-training with Long Captions