# ManyModalQA: Modality Disambiguation and QA over Diverse Inputs

arXiv: https://arxiv.org/abs/2001.08034 · HTML: https://mlsanity.com/paper/2001.08034/

## Key facts

- Cited by 38 papers in the map (data through 7 Oct 2026).
- First posted on arXiv 22 Jan 2020.

## Built on by (most cited first, 30 of 38)

- [MuRAG: Multimodal Retrieval-Augmented Generator for Open Question Answering over Images and Text](https://mlsanity.com/paper/2210.02928/) (2022)
- [MultiModalQA: Complex Question Answering over Text, Tables and Images](https://mlsanity.com/paper/2104.06039/) (2021)
- [MultiBench: Multiscale Benchmarks for Multimodal Representation Learning](https://mlsanity.com/paper/2107.07502/) (2021)
- [WebQA: Multihop and Multimodal QA](https://mlsanity.com/paper/2109.00590/) (2021)
- [MileBench: Benchmarking MLLMs in Long Context](https://mlsanity.com/paper/2404.18532/) (2024)
- [Fine-tuning Multimodal LLMs to Follow Zero-shot Demonstrative Instructions](https://mlsanity.com/paper/2308.04152/) (2023)
- [EHRXQA: A Multi-Modal Question Answering Dataset for Electronic Health Records with Chest X-ray Images](https://mlsanity.com/paper/2310.18652/) (2023)
- [MRAG-Bench: Vision-Centric Evaluation for Retrieval-Augmented Multimodal Models](https://mlsanity.com/paper/2410.08182/) (2024)
- [KramaBench: A Benchmark for AI Systems on Data-to-Insight Pipelines over Data Lakes](https://mlsanity.com/paper/2506.06541/) (2025)
- [Conversational Question Answering on Heterogeneous Sources](https://mlsanity.com/paper/2204.11677/) (2022)
- [Unified Language Representation for Question Answering over Text, Tables, and Images](https://mlsanity.com/paper/2306.16762/) (2023)
- [MuMuQA: Multimedia Multi-Hop News Question Answering via Cross-Media Knowledge Extraction and Grounding](https://mlsanity.com/paper/2112.10728/) (2021)
- [Effect of Visual Extensions on Natural Language Understanding in Vision-and-Language Models](https://mlsanity.com/paper/2104.08066/) (2021)
- [MMHQA-ICL: Multimodal In-context Learning for Hybrid Question Answering over Text, Tables and Images](https://mlsanity.com/paper/2309.04790/) (2023)
- [SnapNTell: Enhancing Entity-Centric Visual Question Answering with Retrieval Augmented Multimodal LLM](https://mlsanity.com/paper/2403.04735/) (2024)
- [DrugEHRQA: A Question Answering Dataset on Structured and Unstructured Electronic Health Records For Medicine Related Queries](https://mlsanity.com/paper/2205.01290/) (2022)
- [Multimodal Conversational AI: A Survey of Datasets and Approaches](https://mlsanity.com/paper/2205.06907/) (2022)
- [A Symmetric Dual Encoding Dense Retrieval Framework for Knowledge-Intensive Visual Question Answering](https://mlsanity.com/paper/2304.13649/) (2023)
- [MPMQA: Multimodal Question Answering on Product Manuals](https://mlsanity.com/paper/2304.09660/) (2023)
- [Pre-Training Multi-Modal Dense Retrievers for Outside-Knowledge Visual Question Answering](https://mlsanity.com/paper/2306.16478/) (2023)
- [MoqaGPT : Zero-Shot Multi-modal Open-domain Question Answering with Large Language Model](https://mlsanity.com/paper/2310.13265/) (2023)
- [cPAPERS: A Dataset of Situated and Multimodal Interactive Conversations in Scientific Papers](https://mlsanity.com/paper/2406.08398/) (2024)
- [Memory-QA: Answering Recall Questions Based on Multimodal Memories](https://mlsanity.com/paper/2509.18436/) (2025)
- [Exploring Hybrid Question Answering via Program-based Prompting](https://mlsanity.com/paper/2402.10812/) (2024)
- [Question Decomposition with Dependency Graphs](https://mlsanity.com/paper/2104.08647/) (2021)
- [cTBLS: Augmenting Large Language Models with Conversational Tables](https://mlsanity.com/paper/2303.12024/) (2023)
- [FCMR: Robust Evaluation of Financial Cross-Modal Multi-Hop Reasoning](https://mlsanity.com/paper/2412.12567/) (2024)
- [OPERA: Harmonizing Task-Oriented Dialogs and Information Seeking Experience](https://mlsanity.com/paper/2206.12449/) (2022)
- [Progressive Evidence Refinement for Open-domain Multimodal Retrieval Question Answering](https://mlsanity.com/paper/2310.09696/) (2023)
- [CT2C-QA: Multimodal Question Answering over Chinese Text, Table and Chart](https://mlsanity.com/paper/2410.21414/) (2024)

## Abstract (arXiv)

We present a new multimodal question answering challenge, ManyModalQA, in which an agent must answer a question by considering three distinct modalities: text, images, and tables. We collect our data by scraping Wikipedia and then utilize crowdsourcing to collect question-answer pairs. Our questions are ambiguous, in that the modality that contains the answer is not easily determined based solely upon the question. To demonstrate this ambiguity, we construct a modality selector (or disambiguator) network, and this model gets substantially lower accuracy on our challenge set, compared to existing datasets, indicating that our questions are more ambiguous. By analyzing this model, we investigate which words in the question are indicative of the modality. Next, we construct a simple baseline ManyModalQA model, which, based on the prediction from the modality selector, fires a corresponding pre-trained state-of-the-art unimodal QA model. We focus on providing the community with a new manymodal evaluation set and only provide a fine-tuning set, with the expectation that existing datasets and approaches will be transferred for most of the training, to encourage low-resource generalization without large, monolithic training sets for each new task. There is a significant gap between our baseline models and human performance; therefore, we hope that this challenge encourages research in end-to-end modality disambiguation and multimodal QA models, as well as transfer learning. Code and data available at: https://github.com/hannandarryl/ManyModalQA