mlsanityData to 7 Oct 2026

WebQA: Multihop and Multimodal QA

arXiv 2109.00590 ↗First posted 1 Sep 2021cs.CL cs.AI cs.CVcited by 109

Not on the map. Why?

Cited by 33 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Scaling Visual Question Answering (VQA) to the open-domain and multi-hop nature of web searches, requires fundamental advances in visual representation learning, knowledge aggregation, and language generation. In this work, we introduce WebQA, a challenging new benchmark that proves difficult for large-scale state-of-the-art models which lack language groundable visual representations for novel objects and the ability to reason, yet trivial for humans. WebQA mirrors the way humans use the web: 1) Ask a question, 2) Choose sources to aggregate, and 3) Produce a fluent language response. This is the behavior we should be expecting from IoT devices and digital assistants. Existing work prefers to assume that a model can either reason about knowledge in images or in text. WebQA includes a secondary text-only QA task to ensure improved visual performance does not come at the cost of language understanding. Our challenge for the community is to create unified multimodal reasoning models that answer questions regardless of the source modality, moving us closer to digital assistants that not only query language knowledge, but also the richer visual online world.

Key facts. Cited by 109 papers in the map (data through 7 Oct 2026). Builds on 29 of its 39 references that are in the map. First posted on arXiv 1 Sep 2021.

Builds on

29 of 39 references.

References · 29

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  5. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  6. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  7. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  8. 2016Aggregated Residual Transformations for Deep Neural Networkscited by 2,540
  9. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  10. 2015VQA: Visual Question Answeringcited by 2,489
  11. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  12. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
17 more
  1. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
  2. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,114
  3. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  4. 2021BARTScore: Evaluating Generated Text as Text Generationcited by 719
  5. 2019Unified Vision-Language Pre-Training for Image Captioning and VQAcited by 494
  6. 2018TVQA: Localized, Compositional Video Question Answeringcited by 463
  7. 2018A Corpus for Reasoning About Natural Language Grounded in Photographscited by 457
  8. 2021Unifying Vision-and-Language Tasks via Text Generationcited by 411
  9. 2018The Web as a Knowledge-base for Answering Complex Questionscited by 409
  10. 2015MovieQA: Understanding Stories in Movies through Question-Answeringcited by 304
  11. 2020Experience Grounds Languagecited by 241
  12. 2017Constructing Datasets for Multi-hop Reading Comprehension Across Documentscited by 190
  13. 2021MultiModalQA: Complex Question Answering over Text, Tables and Imagescited by 177
  14. 2020In Defense of Grid Features for Visual Question Answeringcited by 155
  15. 2018RecipeQA: A Challenge Dataset for Multimodal Comprehension of Cooking Recipescited by 74
  16. 2020ManyModalQA: Modality Disambiguation and QA over Diverse Inputscited by 38
  17. 2021What Ingredients Make for an Effective Crowdsourcing Protocol for Difficult NLU Data Collection Tasks?cited by 23

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 13; 2023: 16; 2024: 20; 2025: 31; 2026: 28.

109 papers.

Sort:

By area:

  1. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,669
  2. 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 702
  3. 2023A Comprehensive Overview of Large Language Modelscited by 639
  4. 2024VisRAG: Vision-based Retrieval-augmented Generation on Multi-modality Documentscited by 237
  5. 2024VLM2Vec: Training Vision-Language Models for Massive Multimodal Embedding Taskscited by 222
  6. 2022MuRAG: Multimodal Retrieval-Augmented Generator for Open Question Answering over Images and Textcited by 216
  7. 2022Re-Imagen: Retrieval-Augmented Text-to-Image Generatorcited by 208
  8. 2024Agent AI: Surveying the Horizons of Multimodal Interactioncited by 167
  9. 2024GME: Improving Universal Multimodal Retrieval by Multimodal LLMscited by 166
  10. 2023UniIR: Training and Benchmarking Universal Multimodal Information Retrieverscited by 165
  11. 2024MM-Embed: Universal Multimodal Retrieval with Multimodal LLMscited by 159
  12. 2022Vision-Language Pre-training: Basics, Recent Advances, and Future Trendscited by 117
38 more
  1. 2023Open-domain Visual Entity Recognition: Towards Recognizing Millions of Wikipedia Entitiescited by 112
  2. 2022PromptCap: Prompt-Guided Task-Aware Image Captioningcited by 104
  3. 2024MileBench: Benchmarking MLLMs in Long Contextcited by 86
  4. 2022K-LITE: Learning Transferable Visual Models with External Knowledgecited by 79
  5. 2023Fine-tuning Multimodal LLMs to Follow Zero-shot Demonstrative Instructionscited by 76
  6. 2023Generative Pre-trained Transformer: A Comprehensive Review on Enabling Technologies, Potential Applications, Emerging Challenges, and Future Directionscited by 55
  7. 2022VL-InterpreT: An Interactive Visualization Tool for Interpreting Vision-Language Transformerscited by 53
  8. 2023EHRXQA: A Multi-Modal Question Answering Dataset for Electronic Health Records with Chest X-ray Imagescited by 53
  9. 2024Visual Haystacks: A Vision-Centric Needle-In-A-Haystack Benchmarkcited by 53
  10. 2024MLLM Is a Strong Reranker: Advancing Multimodal Retrieval-augmented Generation via Knowledge-enhanced Reranking and Noise-injected Trainingcited by 50
  11. 2024MARS: Meaning-Aware Response Scoring for Uncertainty Estimation in Generative LLMscited by 45
  12. 2022Neural Approaches to Conversational Information Retrievalcited by 41
  13. 2024MRAG-Bench: Vision-Centric Evaluation for Retrieval-Augmented Multimodal Modelscited by 41
  14. 2021Yuan 1.0: Large-Scale Pre-trained Language Model in Zero-Shot and Few-Shot Learningcited by 35
  15. 2025Retrieval Augmented Generation and Understanding in Vision: A Survey and New Outlookcited by 32
  16. 2022Enhancing Multi-modal and Multi-hop Question Answering via Structured Knowledge and Unified Retrieval-Generationcited by 28
  17. 2023EDIS: Entity-Driven Image Search over Multimodal Web Contentcited by 27
  18. 2025Modality Curation: Building Universal Embeddings for Advanced Multimodal Information Retrievalcited by 26
  19. 2025FullFront: Benchmarking MLLMs Across the Full Front-End Engineering Workflowcited by 25
  20. 2022Modern Question Answering Datasets and Benchmarks: A Surveycited by 23
  21. 2023End-to-end Knowledge Retrieval with Multi-modal Queriescited by 23
  22. 2024Wings: Learning Multimodal LLMs without Text-only Forgettingcited by 23
  23. 2023Unified Language Representation for Question Answering over Text, Tables, and Imagescited by 22
  24. 2024Goal-guided Generative Prompt Injection Attack on Large Language Modelscited by 22
  25. 2025Retrieval-Augmented Perception: High-Resolution Image Perception Meets Visual RAGcited by 22
  26. 2024HEMM: Holistic Evaluation of Multimodal Foundation Modelscited by 21
  27. 2025From Calibration to Collaboration: LLM Uncertainty Quantification Should Be More Human-Centeredcited by 21
  28. 2022VLC-BERT: Visual Question Answering with Contextualized Commonsense Knowledgecited by 20
  29. 2024SnapNTell: Enhancing Entity-Centric Visual Question Answering with Retrieval Augmented Multimodal LLMcited by 18
  30. 2025UniversalRAG: Retrieval-Augmented Generation over Corpora of Diverse Modalities and Granularitiescited by 18
  31. 2024Document Haystacks: Vision-Language Reasoning Over Piles of 1000+ Documentscited by 17
  32. 2025WebMMU: A Benchmark for Multimodal Multilingual Website Understanding and Code Generationcited by 16
  33. 2023UniS-MMC: Multimodal Classification via Unimodality-supervised Multimodal Contrastive Learningcited by 13
  34. 2025Benchmarking Retrieval-Augmented Generation in Multi-Modal Contextscited by 12
  35. 2025BrowserArena: Evaluating LLM Agents on Real-World Web Navigation Taskscited by 12
  36. 2025MIEB: Massive Image Embedding Benchmarkcited by 11
  37. 2023MPMQA: Multimodal Question Answering on Product Manualscited by 10
  38. 2023Answer Mining from a Pool of Images: Towards Retrieval-Based Visual Question Answeringcited by 10

50 most cited of 109 All 109 (JSON)