mlsanityData to 7 Oct 2026

Multimodal Inverse Cloze Task for Knowledge-based Visual Question Answering

arXiv 2301.04366 ↗First posted 11 Jan 2023cs.CL cs.IR cs.LGcited by 5

Not on the map. Why?

Abstract (arXiv)

We present a new pre-training method, Multimodal Inverse Cloze Task, for Knowledge-based Visual Question Answering about named Entities (KVQAE). KVQAE is a recently introduced task that consists in answering questions about named entities grounded in a visual context using a Knowledge Base. Therefore, the interaction between the modalities is paramount to retrieve information and must be captured with complex fusion models. As these models require a lot of training data, we design this pre-training task from existing work in textual Question Answering. It consists in considering a sentence as a pseudo-question and its context as a pseudo-relevant passage and is extended by considering images near texts in multimodal documents. Our method is applicable to different neural network architectures and leads to a 9% relative-MRR and 15% relative-F1 gain for retrieval and reading comprehension, respectively, over a no-pre-training baseline.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 48 of its 66 references that are in the map. First posted on arXiv 11 Jan 2023.

Builds on

48 of 66 references.

References · 48

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  5. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  6. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  7. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  8. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  9. 2016Layer Normalizationcited by 5,579
  10. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  11. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  12. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,462
36 more
  1. 2015VQA: Visual Question Answeringcited by 2,489
  2. 2017Billion-scale similarity search with GPUscited by 2,477
  3. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  4. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,201
  5. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  6. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
  7. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,114
  8. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,041
  9. 2021Transformers in Vision: A Surveycited by 1,009
  10. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  11. 2016Training Deep Nets with Sublinear Memory Costcited by 825
  12. 2019Latent Retrieval for Weakly Supervised Open Domain Question Answeringcited by 691
  13. 2020KILT: a Benchmark for Knowledge Intensive Language Taskscited by 571
  14. 2021Datasets: A Community Library for Natural Language Processingcited by 526
  15. 2019Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-trainingcited by 451
  16. 2020RocketQA: An Optimized Training Approach to Dense Passage Retrieval for Open-Domain Question Answeringcited by 410
  17. 2020Hard Negative Mixing for Contrastive Learningcited by 399
  18. 2014Memory Networkscited by 354
  19. 2021WIT: Wikipedia-based Image Text Dataset for Multimodal Multilingual Machine Learningcited by 285
  20. 2020Pretrained Transformers for Text Ranking: BERT and Beyondcited by 266
  21. 2021Efficiently Teaching an Effective Dense Retriever with Balanced Topic Aware Samplingcited by 238
  22. 2021Unsupervised Corpus Aware Language Model Pre-training for Dense Passage Retrievalcited by 203
  23. 2021Condenser: a Pre-training Architecture for Dense Retrievalcited by 162
  24. 2020Multimodal Pretraining Unmasked: A Meta-Analysis and a Unified Framework of Vision-and-Language BERTscited by 71
  25. 2020Does my multimodal model learn cross-modal interactions? It's harder to tell than you might think!cited by 61
  26. 2021Weakly-Supervised Visual-Retriever-Reader for Knowledge-based Question Answeringcited by 50
  27. 2021Learning to Retrieve Passages without Supervisioncited by 46
  28. 2021Retrieve Fast, Rerank Smart: Cooperative and Joint Approaches for Improved Cross-Modal Retrievalcited by 27
  29. 2021Passage Retrieval for Outside-Knowledge Visual Question Answeringcited by 25
  30. 2021Reasoning over Vision and Language: Exploring the Benefits of Supplemental Knowledgecited by 24
  31. 2022A Unified End-to-End Retriever-Reader Framework for Knowledge-based VQAcited by 18
  32. 2016Cross-Modal Scene Networkscited by 16
  33. 2022Hypergraph Transformer: Weakly-supervised Multi-hop Reasoning for Knowledge-based Visual Question Answeringcited by 15
  34. 2021Improving and Diagnosing Knowledge-Based Visual Question Answering via Entity Enhanced Knowledge Injectioncited by 8
  35. 2021Efficient Retrieval Optimized Multi-task Learningcited by 3
  36. 2022Modeling Temporal-Modal Entity Graph for Procedural Multimodal Machine Comprehensioncited by 2

Built on by

Cited by year

Citing papers per year: 2023: 2; 2024: 1; 2025: 2.

5 papers.

  1. 2024Cross-modal Retrieval for Knowledge-based Visual Question Answeringcited by 13
  2. 2023Pre-Training Multi-Modal Dense Retrievers for Outside-Knowledge Visual Question Answeringcited by 10
  3. 2025FAR-Net: Multi-Stage Fusion Network with Enhanced Semantic Alignment and Adaptive Reconciliation for Composed Image Retrievalcited by 3
  4. 2023The Contribution of Knowledge in Visiolinguistic Learning: A Survey on Tasks and Challengescited by 2
  5. 2025Augmented Vision-Language Models: A Systematic Reviewcited by 0