mlsanityData to 7 Oct 2026

MAP: Multimodal Uncertainty-Aware Vision-Language Pre-training Model

arXiv 2210.05335 ↗First posted 11 Oct 2022cs.CV cs.CL cs.MMcited by 41

On the map: Multimodal LLMs / vision-language models → Hallucination mitigation in LVLMs

Abstract (arXiv)

Multimodal semantic understanding often has to deal with uncertainty, which means the obtained messages tend to refer to multiple targets. Such uncertainty is problematic for our interpretation, including inter- and intra-modal uncertainty. Little effort has studied the modeling of this uncertainty, particularly in pre-training on unlabeled datasets and fine-tuning in task-specific downstream datasets. In this paper, we project the representations of all modalities as probabilistic distributions via a Probability Distribution Encoder (PDE) by utilizing sequence-level interactions. Compared to the existing deterministic methods, such uncertainty modeling can convey richer multimodal semantic information and more complex relationships. Furthermore, we integrate uncertainty modeling with popular pre-training frameworks and propose suitable pre-training tasks: Distribution-based Vision-Language Contrastive learning (D-VLC), Distribution-based Masked Language Modeling (D-MLM), and Distribution-based Image-Text Matching (D-ITM). The fine-tuned models are applied to challenging downstream tasks, including image-text retrieval, visual question answering, visual reasoning, and visual entailment, and achieve state-of-the-art results.

Key facts. Cited by 41 papers in the map (data through 7 Oct 2026). Placed in Hallucination mitigation in LVLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 36 of its 60 references that are in the map. First posted on arXiv 11 Oct 2022.

Builds on

36 of 60 references.

References · 36

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  6. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  7. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  8. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  9. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  10. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
  11. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  12. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,352
24 more
  1. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,224
  2. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
  3. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
  4. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  5. 2021MDETR -- Modulated Detection for End-to-End Multi-Modal Understandingcited by 811
  6. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervisioncited by 645
  7. 2018A Corpus for Reasoning About Natural Language Grounded in Photographscited by 457
  8. 2021VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Expertscited by 424
  9. 2019Deep Modular Co-Attention Networks for Visual Question Answeringcited by 338
  10. 2020Large-Scale Adversarial Training for Vision-and-Language Representation Learningcited by 327
  11. 2020A Recurrent Vision-and-Language BERT for Navigationcited by 300
  12. 2021An Empirical Study of Training End-to-End Vision-and-Language Transformerscited by 271
  13. 2020UNIMO: Towards Unified-Modal Understanding and Generation via Cross-Modal Contrastive Learningcited by 239
  14. 2022Vision-Language Pre-Training with Triple Contrastive Learningcited by 212
  15. 2021Seeing Out of tHe bOx: End-to-End Pre-training for Vision-Language Representation Learningcited by 187
  16. 2021Probabilistic Embeddings for Cross-Modal Retrievalcited by 150
  17. 2020Data Uncertainty Learning in Face Recognitioncited by 134
  18. 2014Word Representations via Gaussian Embeddingcited by 96
  19. 2021E2E-VLP: End-to-End Vision-Language Pre-training Enhanced by Visual Learningcited by 84
  20. 2019View-Invariant Probabilistic Embedding for Human Posecited by 47
  21. 2018Visual Entailment Task for Visually-Grounded Language Learningcited by 36
  22. 2019Exposing and Correcting the Gender Bias in Image Captioning Datasets and Modelscited by 36
  23. 2021Probabilistic Modeling of Semantic Ambiguity for Scene Graph Generationcited by 35
  24. 2017Multimodal Word Distributionscited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 4; 2024: 14; 2025: 19; 2026: 4.

41 papers.

Sort:

By area:

  1. 2023Improved Probabilistic Image-Text Representationscited by 40
  2. 2023SA-Attack: Improving Adversarial Transferability of Vision-Language Pre-training Models via Self-Augmentationcited by 39
  3. 2024ExACT: Language-guided Conceptual Reasoning and Uncertainty Estimation for Event-based Action Recognition and Morecited by 33
  4. 2024Prompt-A-Video: Prompt Your Video Diffusion Model via Preference-Aligned LLMcited by 22
  5. 2024CRSOT: Cross-Resolution Object Tracking using Unaligned Frame and Event Camerascited by 17
  6. 2025Evaluating Uncertainty and Quality of Vision-Language-Action-enabled Robotscited by 17
  7. 2024Probabilistic Language-Image Pre-Trainingcited by 16
  8. 2024Open-Set Recognition in the Age of Vision-Language Modelscited by 13
  9. 2024Post-hoc Probabilistic Vision-Language Modelscited by 13
  10. 2025PeRL: Permutation-Enhanced Reinforcement Learning for Interleaved Vision-Language Reasoningcited by 11
  11. 2024IDA-VLM: Towards Movie Understanding via ID-Aware Large Vision-Language Modelcited by 10
  12. 2025Dynamic Uncertainty-aware Multimodal Fusion for Outdoor Health Monitoringcited by 10
29 more
  1. 2024Textualized and Feature-based Models for Compound Multimodal Emotion Recognition in the Wildcited by 9
  2. 2025Probabilistic Embeddings for Frozen Vision-Language Models: Uncertainty Quantification with Gaussian Process Latent Variable Modelscited by 9
  3. 2024PIN: A Knowledge-Intensive Dataset for Paired and Interleaved Multimodal Documentscited by 6
  4. 2024UniMat: Unifying Materials Embeddings through Multi-modal Learningcited by 6
  5. 2024Enhancing Multimodal Emotion Recognition through Multi-Granularity Cross-Modal Alignmentcited by 5
  6. 2025Uncertainty-o: One Model-agnostic Framework for Unveiling Uncertainty in Large Multimodal Modelscited by 4
  7. 2023Dynamic Visual Semantic Sub-Embeddings and Fast Re-Rankingcited by 3
  8. 2023TiMix: Text-aware Image Mixing for Effective Vision-Language Pre-trainingcited by 3
  9. 2024Enhancing Emotion Recognition in Incomplete Data: A Novel Cross-Modal Alignment, Reconstruction, and Refinement Frameworkcited by 3
  10. 2025UGG-ReID: Uncertainty-Guided Graph Model for Multi-Modal Object Re-Identificationcited by 3
  11. 2025Entropy Heat-Mapping: Localizing GPT-Based OCR Errors with Sliding-Window Shannon Analysiscited by 2
  12. 2025Hallucination Filtering in Radiology Vision-Language Models Using Discrete Semantic Entropycited by 2
  13. 2025Intra-Class Probabilistic Embeddings for Uncertainty Estimation in Vision-Language Modelscited by 2
  14. 2024Semantics-enhanced Cross-modal Masked Image Modeling for Vision-Language Pre-trainingcited by 1
  15. 2025LongProLIP: A Probabilistic Vision-Language Model with Long Context Textcited by 1
  16. 2025Text-guided multi-stage cross-perception network for medical image segmentationcited by 1
  17. 2025Multi-modal Uncertainty Robust Tree Cover Segmentation For High-Resolution Remote Sensing Imagescited by 1
  18. 2025ProM3E: Probabilistic Masked MultiModal Embedding Model for Ecologycited by 1
  19. 2025Calibrating Uncertainty for Zero-Shot Adversarial CLIPcited by 1
  20. 2026Variational Adapter for Cross-modal Similarity Representationcited by 1
  21. 2024Data Uncertainty-Aware Learning for Multimodal Aspect-based Sentiment Analysiscited by 0
  22. 2025Feature-Based Dual Visual Feature Extraction Model for Compound Multimodal Emotion Recognitioncited by 0
  23. 2025Uncertainty-driven Embedding Convolutioncited by 0
  24. 2025TARS: MinMax Token-Adaptive Preference Strategy for Hallucination Reduction in MLLMscited by 0
  25. 2025Cross Modal Fine-Grained Alignment via Granularity-Aware and Region-Uncertain Modelingcited by 0
  26. 2025Robust and Generalizable GNN Fine-Tuning via Uncertainty-aware Adapter Learningcited by 0
  27. 2026SCoOP: Semantic Consistent Opinion Pooling for Uncertainty Quantification in Multiple Vision-Language Model Systemscited by 0
  28. 2026EmoEUS: Uncertainty Supervision for Multimodal Emotion Recognition in Conversationcited by 0
  29. 2026UG-UMRE: Uncertainty-Guided Modality Augmentation and Distributional Calibration for Unified Multimodal Relation Extractioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Exploiting the Asymmetric Uncertainty Structure of Pre-trained VLMs on the Unit Hypersphere
  2. 2025Training-free Uncertainty Guidance for Complex Visual Tasks with MLLMs
  3. 2024Unveiling Uncertainty: A Deep Dive into Calibration and Performance of Multimodal Large Language Models
  4. 2024Uncertainty-Aware Evaluation for Vision-Language Models
  5. 2026FUSE: Quantifying Uncertainty in Vision-Language Models by Bayesian Fusing Epistemic and Aleatoric Uncertainty
  6. 2026Visual Semantic Entropy: Do Vision Language Models Recognize Visual Ambiguity?
  7. 2025Evaluating Robustness of Vision-Language Models Under Noisy Conditions
  8. 2025The Art of Saying "Maybe": A Conformal Lens for Uncertainty Benchmarking in VLMs