mlsanityData to 14 Oct 2026

MMHQA-ICL: Multimodal In-context Learning for Hybrid Question Answering over Text, Tables and Images

arXiv 2309.04790 ↗First posted 9 Sep 2023cs.CLcited by 18

Not on the map. Why?

Abstract (arXiv)

In the real world, knowledge often exists in a multimodal and heterogeneous form. Addressing the task of question answering with hybrid data types, including text, tables, and images, is a challenging task (MMHQA). Recently, with the rise of large language models (LLM), in-context learning (ICL) has become the most popular way to solve QA problems. We propose MMHQA-ICL framework for addressing this problems, which includes stronger heterogeneous data retriever and an image caption module. Most importantly, we propose a Type-specific In-context Learning Strategy for MMHQA, enabling LLMs to leverage their powerful performance in this task. We are the first to use end-to-end LLM prompting method for this task. Experimental results demonstrate that our framework outperforms all baselines and methods trained on the full dataset, achieving state-of-the-art results under the few-shot setting on the MultimodalQA dataset.

Key facts. Cited by 18 papers in the map (data through 14 Oct 2026). Builds on 28 of its 35 references that are in the map. First posted on arXiv 9 Sep 2023.

Builds on

28 of 35 references.

References · 28

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  3. 2023Visual Instruction Tuningcited by 7,869
  4. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  5. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  6. 2015VQA: Visual Question Answeringcited by 2,506
  7. 2021DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharingcited by 1,254
  8. 2022LaMDA: Language Models for Dialog Applicationscited by 1,220
  9. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,202
  10. 2018Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Taskcited by 1,139
  11. 2023Multimodal Chain-of-Thought Reasoning in Language Modelscited by 752
  12. 2015Compositional Semantic Parsing on Semi-Structured Tablescited by 515
16 more
  1. 2021TAT-QA: A Question Answering Benchmark on a Hybrid of Tabular and Textual Content in Financecited by 408
  2. 2020HybridQA: A Dataset of Multi-Hop Question Answering over Tabular and Textual Datacited by 264
  3. 2022Binding Language Models in Symbolic Languagescited by 228
  4. 2022MuRAG: Multimodal Retrieval-Augmented Generator for Open Question Answering over Images and Textcited by 220
  5. 2021MultiModalQA: Complex Question Answering over Text, Tables and Imagescited by 179
  6. 2023Large Language Models are Versatile Decomposers: Decompose Evidence and Questions for Table-based Reasoningcited by 178
  7. 2022MultiHiertt: Numerical Reasoning over Multi Hierarchical Tabular and Textual Datacited by 121
  8. 2022Large Language Models are few(1)-shot Table Reasonerscited by 116
  9. 2021Turning Tables: Generating Examples from Semi-structured Tables for Endowing Language Models with Reasoning Skillscited by 39
  10. 2020ManyModalQA: Modality Disambiguation and QA over Diverse Inputscited by 38
  11. 2022Enhancing Multi-modal and Multi-hop Question Answering via Structured Knowledge and Unified Retrieval-Generationcited by 28
  12. 2021MuMuQA: Multimedia Multi-Hop News Question Answering via Cross-Media Knowledge Extraction and Groundingcited by 19
  13. 2022Answering Numerical Reasoning Questions in Table-Text Hybrid Contents with Graph-based Encoder and Tree-based Decodercited by 17
  14. 2023S3HQA: A Three-Stage Approach for Multi-hop Text-Table Hybrid Question Answeringcited by 17
  15. 2023Multi-View Graph Representation Learning for Answering Hybrid Numerical Reasoning Questioncited by 10
  16. 2023HopPG: Self-Iterative Program Generation for Multi-Hop Question Answering over Heterogeneous Knowledgecited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 2; 2024: 6; 2025: 7; 2026: 3.

18 papers.

Sort:

By area:

  1. 2025A Survey of Multimodal Retrieval-Augmented Generationcited by 37
  2. 2025Can Multimodal Large Language Models Understand Spatial Relations?cited by 20
  3. 2024TAT-LLM: A Specialized Language Model for Discrete Reasoning over Tabular and Textual Datacited by 19
  4. 2024SymDPO: Boosting In-Context Learning of Large Multimodal Models with Symbol Demonstration Direct Preference Optimizationcited by 10
  5. 2025ContextNav: Towards Agentic Multimodal In-Context Learningcited by 10
  6. 2024Exploring Hybrid Question Answering via Program-based Promptingcited by 8
  7. 2023TableQAKit: A Comprehensive and Practical Toolkit for Table-based Question Answeringcited by 7
  8. 2024SAIL: Sample-Centric In-Context Learning for Document Information Extractioncited by 6
  9. 2023HRoT: Hybrid prompt strategy and Retrieval of Thought for Table-Text Hybrid Question Answeringcited by 5
  10. 2025RAGOps: Operating and Managing Retrieval-Augmented Generation Pipelinescited by 5
  11. 2025Table Question Answering in the Era of Large Language Models: A Comprehensive Survey of Tasks, Methods, and Evaluationcited by 5
  12. 2024Self-adaptive Multimodal Retrieval-Augmented Generationcited by 4
6 more
  1. 2024An Entailment Tree Generation Approach for Multimodal Multi-Hop Question Answering with Mixture-of-Experts and Iterative Feedback Mechanismcited by 3
  2. 2025Agentic Design Review Systemcited by 2
  3. 2025MMAPG: A Training-Free Framework for Multimodal Multi-hop Question Answering via Adaptive Planning Graphscited by 1
  4. 2026Geometry-Aware Hallucination Detection in Large Language Modelscited by 0
  5. 2026Human-Inspired Context-Selective Multimodal Memory for Social Robotscited by 0
  6. 2026ColGraphRAG: Late-Interaction Evidence Retrieval for Multimodal GraphRAGcited by 0