mlsanityData to 7 Oct 2026

MultiModalQA: Complex Question Answering over Text, Tables and Images

arXiv 2104.06039 ↗First posted 13 Apr 2021cs.CL cs.AI cs.LGcited by 177

Not on the map. Why?

Cited by 39 in Retrieval-augmented generation, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

When answering complex questions, people can seamlessly combine information from visual, textual and tabular sources. While interest in models that reason over multiple pieces of evidence has surged in recent years, there has been relatively little work on question answering models that reason across multiple modalities. In this paper, we present MultiModalQA(MMQA): a challenging question answering dataset that requires joint reasoning over text, tables and images. We create MMQA using a new framework for generating complex multi-modal questions at scale, harvesting tables from Wikipedia, and attaching images and text paragraphs using entities that appear in each table. We then define a formal language that allows us to take questions that can be answered from a single modality, and combine them to generate cross-modal questions. Last, crowdsourcing workers take these automatically-generated questions and rephrase them into more fluent language. We create 29,918 questions through this procedure, and empirically demonstrate the necessity of a multi-modal multi-hop approach to solve our task: our multi-hop model, ImplicitDecomp, achieves an average F1of 51.7 over cross-modal questions, substantially outperforming a strong baseline that achieves 38.2 F1, but still lags significantly behind human performance, which is at 90.1 F1

Key facts. Cited by 177 papers in the map (data through 7 Oct 2026). Builds on 36 of its 41 references that are in the map. First posted on arXiv 13 Apr 2021.

Builds on

36 of 41 references.

References · 36

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  5. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  6. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  7. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  8. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  9. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  10. 2015VQA: Visual Question Answeringcited by 2,489
  11. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  12. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
24 more
  1. 2019Towards VQA Models That Can Readcited by 1,696
  2. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  3. 2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,445
  4. 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,310
  5. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
  6. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  7. 2019DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphscited by 883
  8. 2018From Recognition to Cognition: Visual Commonsense Reasoningcited by 553
  9. 2015Compositional Semantic Parsing on Semi-Structured Tablescited by 509
  10. 2020TAPAS: Weakly Supervised Table Parsing via Pre-trainingcited by 432
  11. 2019TabFact: A Large-scale Dataset for Table-based Fact Verificationcited by 425
  12. 2018The Web as a Knowledge-base for Answering Complex Questionscited by 409
  13. 2019PlotQA: Reasoning over Scientific Plotscited by 317
  14. 201912-in-1: Multi-Task Vision and Language Representation Learningcited by 278
  15. 2020HybridQA: A Dataset of Multi-Hop Question Answering over Tabular and Textual Datacited by 263
  16. 2019UNITER: UNiversal Image-TExt Representation Learningcited by 225
  17. 2017Constructing Datasets for Multi-hop Reading Comprehension Across Documentscited by 190
  18. 2019Are We Modeling the Task or the Annotator? An Investigation of Annotator Bias in Natural Language Understanding Datasetscited by 173
  19. 2020Open Question Answering over Tables and Textcited by 163
  20. 2019Learning to Retrieve Reasoning Paths over Wikipedia Graph for Question Answeringcited by 147
  21. 2015Yin and Yang: Balancing and Answering Binary Visual Questionscited by 109
  22. 2019Compositional Questions Do Not Necessitate Multi-hop Reasoningcited by 87
  23. 2018How Much Reading Does Reading Comprehension Require? A Critical Investigation of Popular Benchmarkscited by 86
  24. 2020ManyModalQA: Modality Disambiguation and QA over Diverse Inputscited by 38

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 12; 2022: 21; 2023: 27; 2024: 36; 2025: 45; 2026: 36.

177 papers.

Sort:

By area:

  1. 2021MuSiQue: Multihop Questions via Single-hop Question Compositioncited by 893
  2. 2022Multimodal Learning with Transformers: A Surveycited by 399
  3. 2022Dynamic Prompt Learning via Policy Gradient for Semi-structured Mathematical Reasoningcited by 347
  4. 2022UnifiedSKG: Unifying and Multi-Tasking Structured Knowledge Grounding with Text-to-Text Language Modelscited by 231
  5. 2022Binding Language Models in Symbolic Languagescited by 224
  6. 2022MuRAG: Multimodal Retrieval-Augmented Generator for Open Question Answering over Images and Textcited by 216
  7. 2022Learning to Answer Questions in Dynamic Audio-Visual Scenarioscited by 188
  8. 2023SlideVQA: A Dataset for Document Visual Question Answering on Multiple Imagescited by 175
  9. 2024FlashRAG: A Modular Toolkit for Efficient Retrieval-Augmented Generation Researchcited by 165
  10. 2021SituatedQA: Incorporating Extra-Linguistic Contexts into QAcited by 145
  11. 2021MultiBench: Multiscale Benchmarks for Multimodal Representation Learningcited by 143
  12. 2022MultiHiertt: Numerical Reasoning over Multi Hierarchical Tabular and Textual Datacited by 120
38 more
  1. 2022Large Language Models are few(1)-shot Table Reasonerscited by 116
  2. 2023Table Meets LLM: Can Large Language Models Understand Structured Table Data? A Benchmark and Empirical Studycited by 116
  3. 2021WebQA: Multihop and Multimodal QAcited by 109
  4. 2022CommonsenseQA 2.0: Exposing the Limits of AI through Gamificationcited by 108
  5. 2021QA Dataset Explosion: A Taxonomy of NLP Resources for Question Answering and Reading Comprehensioncited by 91
  6. 2023Answering Questions by Meta-Reasoning over Multiple Chains of Thoughtcited by 87
  7. 2024MileBench: Benchmarking MLLMs in Long Contextcited by 86
  8. 2024MMToM-QA: Multimodal Theory of Mind Question Answeringcited by 84
  9. 2023Fine-tuning Multimodal LLMs to Follow Zero-shot Demonstrative Instructionscited by 76
  10. 2022OpenCQA: Open-ended Question Answering with Chartscited by 60
  11. 2021MATE: Multi-view Attention for Table Transformer Efficiencycited by 54
  12. 2023EHRXQA: A Multi-Modal Question Answering Dataset for Electronic Health Records with Chest X-ray Imagescited by 53
  13. 2024Visual Haystacks: A Vision-Centric Needle-In-A-Haystack Benchmarkcited by 53
  14. 2023ECG-QA: A Comprehensive Question Answering Dataset Combined With Electrocardiogramcited by 52
  15. 2022A Survey on Text-to-SQL Parsing: Concepts, Methods, and Future Directionscited by 51
  16. 2024MLLM Is a Strong Reranker: Advancing Multimodal Retrieval-augmented Generation via Knowledge-enhanced Reranking and Noise-injected Trainingcited by 50
  17. 2022PACIFIC: Towards Proactive Conversational Question Answering over Tabular and Textual Data in Financecited by 45
  18. 2022QAMPARI: An Open-domain Question Answering Benchmark for Questions with Many Answers from Multiple Paragraphscited by 42
  19. 2024On the Multi-turn Instruction Following for Conversational Web Agentscited by 42
  20. 2025Ask in Any Modality: A Comprehensive Survey on Multimodal Retrieval-Augmented Generationcited by 42
  21. 2024MRAG-Bench: Vision-Centric Evaluation for Retrieval-Augmented Multimodal Modelscited by 41
  22. 2021Turning Tables: Generating Examples from Semi-structured Tables for Endowing Language Models with Reasoning Skillscited by 39
  23. 2025A Survey of Multimodal Retrieval-Augmented Generationcited by 37
  24. 2025Representation Learning for Tabular Data: A Comprehensive Surveycited by 36
  25. 2025KramaBench: A Benchmark for AI Systems on Data-to-Insight Pipelines over Data Lakescited by 30
  26. 2022Enhancing Multi-modal and Multi-hop Question Answering via Structured Knowledge and Unified Retrieval-Generationcited by 28
  27. 2022Conversational Question Answering on Heterogeneous Sourcescited by 26
  28. 2024M3SciQA: A Multi-Modal Multi-Document Scientific QA Benchmark for Evaluating Foundation Modelscited by 26
  29. 2025Toward Generalizable Evaluation in the LLM Era: A Survey Beyond Benchmarkscited by 26
  30. 2024Evaluating LLMs' Mathematical Reasoning in Financial Document Question Answeringcited by 25
  31. 2025MM-PoisonRAG: Disrupting Multimodal RAG with Local and Global Poisoning Attackscited by 24
  32. 2023TabLib: A Dataset of 627M Tables with Contextcited by 23
  33. 2023Unified Language Representation for Question Answering over Text, Tables, and Imagescited by 22
  34. 2023DialogStudio: Towards Richest and Most Diverse Unified Dataset Collection for Conversational AIcited by 21
  35. 2021MuMuQA: Multimedia Multi-Hop News Question Answering via Cross-Media Knowledge Extraction and Groundingcited by 19
  36. 2023MMHQA-ICL: Multimodal In-context Learning for Hybrid Question Answering over Text, Tables and Imagescited by 18
  37. 2024SnapNTell: Enhancing Entity-Centric Visual Question Answering with Retrieval Augmented Multimodal LLMcited by 18
  38. 2022DrugEHRQA: A Question Answering Dataset on Structured and Unstructured Electronic Health Records For Medicine Related Queriescited by 17

50 most cited of 177 All 177 (JSON)