mlsanityData to 7 Oct 2026

Synthetic Multimodal Question Generation

arXiv 2407.02233 ↗First posted 2 Jul 2024cs.CL cs.AI cs.LGcited by 8

On the map: Retrieval-augmented generation → Multimodal RAG

Abstract (arXiv)

Multimodal Retrieval Augmented Generation (MMRAG) is a powerful approach to question-answering over multimodal documents. A key challenge with evaluating MMRAG is the paucity of high-quality datasets matching the question styles and modalities of interest. In light of this, we propose SMMQG, a synthetic data generation framework. SMMQG leverages interplay between a retriever, large language model (LLM) and large multimodal model (LMM) to generate question and answer pairs directly from multimodal documents, with the questions conforming to specified styles and modalities. We use SMMQG to generate an MMRAG dataset of 1024 questions over Wikipedia documents and evaluate state-of-the-art models using it, revealing insights into model performance that are attainable only through style- and modality-specific evaluation data. Next, we measure the quality of data produced by SMMQG via a human study. We find that the quality of SMMQG-generated synthetic data is on par with the quality of the crowdsourced benchmark MMQA and that downstream evaluation results using both datasets strongly concur.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Placed in Multimodal RAG, one of 15 directions in Retrieval-augmented generation. Builds on 34 of its 52 references that are in the map. First posted on arXiv 2 Jul 2024.

Builds on

34 of 52 references.

Closest ancestors · 2

Same direction, newest first

  1. 2022Retrieval Augmented Visual Question Answering with Outside Knowledge7 Oct
  2. 2022MuRAG: Multimodal Retrieval-Augmented Generator for Open Question Answering over Images and Text6 Oct

Other references · 32

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  5. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  6. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  7. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  8. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  9. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  10. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  11. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  12. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
20 more
  1. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  2. 2022Text Embeddings by Weakly-Supervised Contrastive Pre-trainingcited by 1,216
  3. 2022Reproducible scaling laws for contrastive language-image learningcited by 1,163
  4. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,114
  5. 2019DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphscited by 883
  6. 2024Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Modelscited by 431
  7. 2020Asking and Answering Questions to Evaluate the Factual Consistency of Summariescited by 376
  8. 2020FEQA: A Question Answering Evaluation Framework for Faithfulness Assessment in Abstractive Summarizationcited by 292
  9. 2020HybridQA: A Dataset of Multi-Hop Question Answering over Tabular and Textual Datacited by 263
  10. 2023Can Pre-trained Vision and Language Models Answer Visual Information-Seeking Questions?cited by 203
  11. 2021MultiModalQA: Complex Question Answering over Text, Tables and Imagescited by 177
  12. 2020Training Question Answering Models From Synthetic Datacited by 95
  13. 2023Retrieving Multimodal Information for Augmented Generation: A Surveycited by 84
  14. 2020End-to-End Synthetic Data Generation for Domain Adaptation of Question Answering Systemscited by 65
  15. 2022Universal Vision-Language Dense Retrieval: Learning A Unified Representation Space for Multi-Modal Retrievalcited by 57
  16. 2023Prompt2Model: Generating Deployable Models from Natural Language Instructionscited by 40
  17. 2020Unsupervised Multi-hop Question Answering by Question Generationcited by 38
  18. 2021Do Question Answering Modeling Improvements Hold Across Benchmarks?cited by 20
  19. 2020Generating Natural Questions from Images for Multimodal Assistantscited by 6
  20. 2023MultiQG-TI: Towards Question Generation from Multi-modal Sourcescited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 4; 2026: 2.

8 papers.

  1. 2024ProMQA: Question Answering Dataset for Multimodal Procedural Activity Understandingcited by 8
  2. 2024FM2DS: Few-Shot Multimodal Multihop Data Synthesis with Knowledge Distillation for Question Answeringcited by 7
  3. 2025Benchmarking Multimodal RAG through a Chart-based Document Question-Answering Generation Frameworkcited by 7
  4. 2025BioMol-MQA: A Multi-Modal Question Answering Dataset For LLM Reasoning Over Bio-Molecular Interactionscited by 4
  5. 2025HopWeaver: Cross-Document Synthesis of High-Quality and Authentic Multi-Hop Questionscited by 2
  6. 2026DrawSim-PD: Simulating Student Science Drawings to Support NGSS-Aligned Teacher Diagnostic Reasoningcited by 1
  7. 2026BioGraphletQA: Knowledge-Anchored Generation of Complex QA Datasetscited by 1
  8. 2025Spatial Knowledge Graph-Guided Multimodal Synthesiscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024SK-VQA: Synthetic Knowledge Generation at Scale for Training Context-Augmented Multimodal LLMs
  2. 2025MRAMG-Bench: A Comprehensive Benchmark for Advancing Multimodal Retrieval-Augmented Multimodal Generation
  3. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehension
  4. 2024SimRAG: Self-Improving Retrieval-Augmented Generation for Adapting Large Language Models to Specialized Domains
  5. 2026MKG-RAG-Bench: Benchmarking Retrieval in Multimodal Knowledge Graph-Augmented Generation
  6. 2026Doc-REFRAG: Rethinking Multimodal Document Retrieval-Augmented Generation
  7. 2025Ask in Any Modality: A Comprehensive Survey on Multimodal Retrieval-Augmented Generation
  8. 2025MTRAG: A Multi-Turn Conversational Benchmark for Evaluating Retrieval-Augmented Generation Systems