mlsanityData to 14 Oct 2026

Bayesian Data Reweighting Improves Multimodal Retrieval for Knowledge-Based Visual Question Answering

arXiv 2608.02907 ↗First posted 3 Aug 2026cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Multimodal retrievers are essential for knowledge-based visual question answering, where they retrieve external evidence for image-question pairs. However, existing contrastive training methods typically treat all unmatched query-document pairs as equally informative negatives, which is problematic because many unmatched documents may still be semantically relevant or partially useful. We propose Bayesian Data Reweighting, a probabilistic framework that models query-document importance as latent variables and adaptively infers posterior weights to downweight likely false negatives. With closed-form posterior updates under conjugate priors and stochastic EM optimization, our method consistently improves retrieval accuracy across three retrievers and seven knowledge-based VQA benchmarks.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 30 of its 39 references that are in the map. First posted on arXiv 3 Aug 2026.

Builds on

30 of 39 references.

References · 30

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  3. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  4. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  5. 2015FaceNet: A Unified Embedding for Face Recognition and Clusteringcited by 2,849
  6. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,121
  7. 2016ConceptNet 5.5: An Open Multilingual Graph of General Knowledgecited by 1,067
  8. 2020Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrievalcited by 849
  9. 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 710
  10. 2018Learning to Reweight Examples for Robust Deep Learningcited by 675
  11. 2020Contrastive Learning with Hard Negative Samplescited by 628
  12. 2018From Recognition to Cognition: Visual Commonsense Reasoningcited by 556
18 more
  1. 2020Hard Negative Mixing for Contrastive Learningcited by 402
  2. 2018Not All Samples Are Created Equal: Deep Learning with Importance Samplingcited by 328
  3. 2017Sampling Matters in Deep Embedding Learningcited by 285
  4. 2024VLM2Vec: Training Vision-Language Models for Massive Multimodal Embedding Taskscited by 228
  5. 2023Can Pre-trained Vision and Language Models Answer Visual Information-Seeking Questions?cited by 207
  6. 2023UniIR: Training and Benchmarking Universal Multimodal Information Retrieverscited by 166
  7. 2024MM-Embed: Universal Multimodal Retrieval with Multimodal LLMscited by 161
  8. 2022REVEAL: Retrieval-Augmented Visual-Language Pre-Training with Multi-Source Multimodal Knowledge Memorycited by 123
  9. 2020Boosting Contrastive Self-Supervised Learning with False Negative Cancellationcited by 119
  10. 2023Encyclopedic VQA: Visual questions about detailed properties of fine-grained categoriescited by 107
  11. 2022Retrieval Augmented Visual Question Answering with Outside Knowledgecited by 94
  12. 2024Wiki-LLaVA: Hierarchical Retrieval-Augmented Generation for Multimodal LLMscited by 80
  13. 2024EchoSight: Advancing Visual-Language Models with Wiki Knowledgecited by 62
  14. 2019Hardness-Aware Deep Metric Learningcited by 55
  15. 2021Incremental False Negative Detection for Contrastive Learningcited by 48
  16. 2024PreFLMR: Scaling Up Fine-Grained Late-Interaction Multi-modal Retrieverscited by 42
  17. 2025Retrieval-Augmented Visual Question Answering via Built-in Autoregressive Search Enginescited by 13
  18. 2025Recurrence-Enhanced Vision-and-Language Transformers for Robust Multimodal Document Retrievalcited by 13

Built on by

Not cited yet.