# Evolution of Multimodal Question Answering: From Modality-Adaptive Extraction to Unified Language Representation

arXiv: https://arxiv.org/abs/2609.08896 · HTML: https://mlsanity.com/paper/2609.08896/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 21 of its 26 references that are in the map.
- First posted on arXiv 8 Sep 2026.

## Builds on (most cited first, 21 of 21)

- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [The Curious Case of Neural Text Degeneration](https://mlsanity.com/paper/1904.09751/) (2019)
- [MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models](https://mlsanity.com/paper/2304.10592/) (2023)
- [Hierarchical Neural Story Generation](https://mlsanity.com/paper/1805.04833/) (2018)
- [Kosmos-2: Grounding Multimodal Large Language Models to the World](https://mlsanity.com/paper/2306.14824/) (2023)
- [PaLI: A Jointly-Scaled Multilingual Language-Image Model](https://mlsanity.com/paper/2209.06794/) (2022)
- [CoQA: A Conversational Question Answering Challenge](https://mlsanity.com/paper/1808.07042/) (2018)
- [Language Is Not All You Need: Aligning Perception with Language Models](https://mlsanity.com/paper/2302.14045/) (2023)
- [QuAC : Question Answering in Context](https://mlsanity.com/paper/1808.07036/) (2018)
- [MultiModalQA: Complex Question Answering over Text, Tables and Images](https://mlsanity.com/paper/2104.06039/) (2021)
- [Open-Retrieval Conversational Question Answering](https://mlsanity.com/paper/2005.11364/) (2020)
- [Multimodal Reasoning with Multimodal Knowledge Graph](https://mlsanity.com/paper/2406.02030/) (2024)
- [Turning Tables: Generating Examples from Semi-structured Tables for Endowing Language Models with Reasoning Skills](https://mlsanity.com/paper/2107.07261/) (2021)
- [ManyModalQA: Modality Disambiguation and QA over Diverse Inputs](https://mlsanity.com/paper/2001.08034/) (2020)
- [Enhancing Multi-modal and Multi-hop Question Answering via Structured Knowledge and Unified Retrieval-Generation](https://mlsanity.com/paper/2212.08632/) (2022)
- [Unified Language Representation for Question Answering over Text, Tables, and Images](https://mlsanity.com/paper/2306.16762/) (2023)
- [Retrieval-augmented Multi-modal Chain-of-Thoughts Reasoning for Large Language Models](https://mlsanity.com/paper/2312.01714/) (2023)
- [MORE: Multi-mOdal REtrieval Augmented Generative Commonsense Reasoning](https://mlsanity.com/paper/2402.13625/) (2024)
- [Multimodal Graph Transformer for Multimodal Question Answering](https://mlsanity.com/paper/2305.00581/) (2023)

## Abstract (arXiv)

The rapid growth of multimodal data has intensified the need for question answering (QA) systems capable of reasoning across heterogeneous sources such as text, tables, and images. In this paper, we present a comprehensive methodological comparison of three influential frameworks, namely Multimodal Adaptive Extraction (MAE), Solar, and UniMMQA, tracing the evolution of multimodal question answering from modality-adaptive pipelines to fully unified architectures. We examine how each approach models cross-modal interactions, transforms heterogeneous inputs, and performs reasoning, highlighting key design differences in modality representation, reasoning, and answer generation. Our analysis demonstrates a clear shift from explicit modality-specific processing toward unified text-centric formulations enabled by pre-trained language models (PLMs). Empirical comparisons across benchmark datasets show that this transition leads to substantial improvements in both Exact Match (EM) and F1-Scores, with UniMMQA achieving the most consistent and scalable performance. Despite these advances, we identify persistent challenges, including information loss during modality transformation, error propagation in multi-stage pipelines, and limitations in capturing fine-grained cross-modal dependencies. Overall, this study provides a deeper understanding of current design trends and offers insights into the future direction of unified multimodal reasoning systems.