mlsanityData to 7 Oct 2026

SK-VQA: Synthetic Knowledge Generation at Scale for Training Context-Augmented Multimodal LLMs

arXiv 2406.19593 ↗First posted 28 Jun 2024cs.CL cs.CVcited by 12

On the map: Multimodal LLMs / vision-language models → Document understanding and multimodal RAG

also in 1 more
Abstract (arXiv)

Multimodal retrieval augmented generation (RAG) plays a crucial role in domains such as knowledge-based visual question answering (KB-VQA), where external knowledge is needed to answer a question. However, existing multimodal LLMs (MLLMs) are not designed for context-augmented generation, limiting their effectiveness in such tasks. While synthetic data generation has recently gained attention for training MLLMs, its application for context-augmented generation remains underexplored. To address this gap, we introduce SK-VQA, a large-scale synthetic multimodal dataset containing over 2 million visual question-answer pairs, each associated with context documents containing information necessary to determine the final answer. Compared to previous datasets, SK-VQA contains 11x more unique questions, exhibits greater domain diversity, and covers a broader spectrum of image sources. Through human evaluations, we confirm the high quality of the generated question-answer pairs and their contextual relevance. Extensive experiments show that SK-VQA serves both as a challenging KB-VQA benchmark and as an effective training resource for adapting MLLMs to context-augmented generation. Our results further indicate that models trained on SK-VQA demonstrate enhanced generalization in both context-aware VQA and multimodal RAG settings. SK-VQA is publicly available via Hugging Face Hub.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Placed in Document understanding and multimodal RAG, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 49 of its 58 references that are in the map. First posted on arXiv 28 Jun 2024.

Builds on

49 of 58 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024Wiki-LLaVA: Hierarchical Retrieval-Augmented Generation for Multimodal LLMs23 Apr
  2. 2022REVEAL: Retrieval-Augmented Visual-Language Pre-Training with Multi-Source Multimodal Knowledge Memory10 Dec
  3. 2022Retrieval-Augmented Multimodal Language Modeling22 Nov

Other references · 46

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  4. 2023GPT-4 Technical Reportcited by 18,132
  5. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  6. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  7. 2023Visual Instruction Tuningcited by 7,792
  8. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  9. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  10. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  11. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  12. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,595
34 more
  1. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  2. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
  3. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
  4. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,114
  5. 2022BERTopic: Neural topic modeling with a class-based TF-IDF procedurecited by 779
  6. 2023In-Context Retrieval-Augmented Language Modelscited by 694
  7. 2024A Survey on Hallucination in Large Vision-Language Modelscited by 352
  8. 2023The Curse of Recursion: Training on Generated Data Makes Models Forgetcited by 343
  9. 2022Is synthetic data from generative models ready for image recognition?cited by 320
  10. 2023Analyzing and Mitigating Object Hallucination in Large Vision-Language Modelscited by 314
  11. 2020Google Landmarks Dataset v2 -- A Large-Scale Benchmark for Instance-Level Recognition and Retrievalcited by 305
  12. 2022Generate rather than Retrieve: Large Language Models are Strong Context Generatorscited by 296
  13. 2023Improving Text Embeddings with Large Language Modelscited by 295
  14. 2021WIT: Wikipedia-based Image Text Dataset for Multimodal Multilingual Machine Learningcited by 285
  15. 2023Effective Data Augmentation With Diffusion Modelscited by 282
  16. 2022MuRAG: Multimodal Retrieval-Augmented Generator for Open Question Answering over Images and Textcited by 216
  17. 2023G-LLaVA: Solving Geometric Problem with Multi-Modal Large Language Modelcited by 208
  18. 2023Can Pre-trained Vision and Language Models Answer Visual Information-Seeking Questions?cited by 203
  19. 2021Benchmarking Representation Learning for Natural World Image Collectionscited by 167
  20. 2023UniIR: Training and Benchmarking Universal Multimodal Information Retrieverscited by 165
  21. 2024Long-form factuality in large language modelscited by 146
  22. 2024Is Model Collapse Inevitable? Breaking the Curse of Recursion by Accumulating Real and Synthetic Datacited by 143
  23. 2023Scaling Laws of Synthetic Images for Model Training ... for Nowcited by 107
  24. 2023Encyclopedic VQA: Visual questions about detailed properties of fine-grained categoriescited by 105
  25. 2022Tomayto, Tomahto. Beyond Token-level Answer Equivalence for Question Answering Evaluationcited by 66
  26. 2021Human-Adversarial Visual Question Answeringcited by 57
  27. 2023SocialCounterfactuals: Probing and Mitigating Intersectional Social Biases in Vision-Language Models with Counterfactual Examplescited by 53
  28. 2023Re-ViLM: Retrieval-Augmented Visual Language Model for Zero and Few-Shot Image Captioningcited by 41
  29. 2023COCO-Counterfactuals: Automatically Constructed Counterfactual Examples for Image-Text Pairscited by 39
  30. 2023Dataset Interfaces: Diagnosing Model Failures Using Controllable Counterfactual Generationcited by 35
  31. 2024DesignQA: A Multimodal Benchmark for Evaluating Large Language Models' Understanding of Engineering Documentationcited by 30
  32. 2022NeuroCounterfactuals: Beyond Minimal-Edit Counterfactuals for Richer Data Augmentationcited by 23
  33. 2024SnapNTell: Enhancing Entity-Centric Visual Question Answering with Retrieval Augmented Multimodal LLMcited by 18
  34. 2024Cross-modal Retrieval for Knowledge-based Visual Question Answeringcited by 13

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 6; 2026: 6.

12 papers.

  1. 2025Ask in Any Modality: A Comprehensive Survey on Multimodal Retrieval-Augmented Generationcited by 42
  2. 2025Visual-RAG: Benchmarking Text-to-Image Retrieval Augmented Generation for Visual Knowledge Intensive Queriescited by 16
  3. 2025Traveling Across Languages: Benchmarking Cross-Lingual Consistency in Multimodal LLMscited by 3
  4. 2025Learning from Reasoning Failures via Synthetic Data Generationcited by 2
  5. 2026Learning to Search: A Decision-Based Agent for Knowledge-Based Visual Question Answeringcited by 2
  6. 202620/20 Vision Language Models: A Prescription for Better VLMs through Data Curation Alonecited by 2
  7. 2026Benchmarking Deflection and Hallucination in Large Vision-Language Modelscited by 1
  8. 2025Agent-Centric Projection of Prompting Techniques and Implications for Synthetic Training Data for Large Language Modelscited by 0
  9. 2025Augmented Vision-Language Models: A Systematic Reviewcited by 0
  10. 2026LITTA: Late-Interaction and Test-Time Alignment for Visually-Grounded Multimodal Retrievalcited by 0
  11. 2026Identifying and Resolving Pitfalls of Knowledge-Based VQA Benchmarks: Auditing, Repairing, and Augmentingcited by 0
  12. 2026VLD-RAG: Agentic Vision-Language Retrieval-Augmented Generation for Long, Visually-Rich Multi-Page Documentscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025mKG-RAG: Leveraging Multimodal Knowledge Graphs in Retrieval-Augmented Generation for Knowledge-intensive VQA
  2. 2025Fine-Grained Knowledge Structuring and Retrieval for Visual Question Answering
  3. 2024Enhanced Multimodal RAG-LLM for Accurate Visual Question Answering
  4. 2024Benchmarking Multimodal Retrieval Augmented Generation with Dynamic VQA Dataset and Self-adaptive Planning Agent
  5. 2025ReAG: Reasoning-Augmented Generation for Knowledge-based Visual Question Answering
  6. 2024Towards Flexible Evaluation for Generative Visual Question Answering
  7. 2024mR2AG: Multimodal Retrieval-Reflection-Augmented Generation for Knowledge-Based VQA
  8. 2024EchoSight: Advancing Visual-Language Models with Wiki Knowledge