mlsanityData to 13 Oct 2026

SPIQA: A Dataset for Multimodal Question Answering on Scientific Papers

arXiv 2407.09413 ↗First posted 12 Jul 2024cs.CL cs.AI cs.CVcited by 75

Not on the map. Why?

Abstract (arXiv)

Seeking answers to questions within long scientific research articles is a crucial area of study that aids readers in quickly addressing their inquiries. However, existing question-answering (QA) datasets based on scientific papers are limited in scale and focus solely on textual content. We introduce SPIQA (Scientific Paper Image Question Answering), the first large-scale QA dataset specifically designed to interpret complex figures and tables within the context of scientific research articles across various domains of computer science. Leveraging the breadth of expertise and ability of multimodal large language models (MLLMs) to understand figures, we employ automatic and manual curation to create the dataset. We craft an information-seeking task on interleaved images and text that involves multiple images covering plots, charts, tables, schematic diagrams, and result visualizations. SPIQA comprises 270K questions divided into training, validation, and three different evaluation splits. Through extensive experiments with 12 prominent foundational models, we evaluate the ability of current multimodal systems to comprehend the nuanced aspects of research articles. Additionally, we propose a Chain-of-Thought (CoT) evaluation strategy with in-context retrieval that allows fine-grained, step-by-step assessment and improves model performance. We further explore the upper bounds of performance enhancement with additional textual information, highlighting its promising potential for future research and the dataset's impact on revolutionizing how we interact with scientific literature.

Key facts. Cited by 75 papers in the map (data through 13 Oct 2026). Builds on 59 of its 88 references that are in the map. First posted on arXiv 12 Jul 2024.

Builds on

59 of 88 references.

References · 59

  1. 2023GPT-4 Technical Reportcited by 18,278
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,979
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,492
  4. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,480
  5. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 5,021
  6. 2023Improved Baselines with Visual Instruction Tuningcited by 4,389
  7. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,133
  8. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,716
  9. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,412
  10. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,197
  11. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,925
  12. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,864
38 more
  1. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,228
  2. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,133
  3. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,035
  4. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,595
  5. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,461
  6. 2019PubMedQA: A Dataset for Biomedical Research Question Answeringcited by 1,281
  7. 2016A Diagram Is Worth A Dozen Imagescited by 796
  8. 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 706
  9. 2017The NarrativeQA Reading Comprehension Challengecited by 592
  10. 2024MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems?cited by 580
  11. 2015Compositional Semantic Parsing on Semi-Structured Tablescited by 513
  12. 2021A Dataset of Information-Seeking Questions and Answers Anchored in Research Paperscited by 478
  13. 2019TabFact: A Large-scale Dataset for Table-based Fact Verificationcited by 426
  14. 2018QuAC : Question Answering in Contextcited by 380
  15. 2021Inter-GPS: Interpretable Geometry Problem Solving with Formal Language and Symbolic Reasoningcited by 378
  16. 2018DVQA: Understanding Data Visualizations via Question Answeringcited by 339
  17. 2016NewsQA: A Machine Comprehension Datasetcited by 330
  18. 2024InternLM-XComposer2: Mastering Free-form Text-Image Composition and Comprehension in Vision-Language Large Modelcited by 326
  19. 2019PlotQA: Reasoning over Scientific Plotscited by 317
  20. 2017FigureQA: An Annotated Figure Dataset for Visual Reasoningcited by 316
  21. 2023LongLoRA: Efficient Fine-tuning of Long-Context Large Language Modelscited by 244
  22. 2021GeoQA: A Geometric Question Answering Benchmark Towards Multimodal Numerical Reasoningcited by 225
  23. 2017Constructing Datasets for Multi-hop Reading Comprehension Across Documentscited by 190
  24. 2021FeTaQA: Free-form Table Question Answeringcited by 175
  25. 2023SlideVQA: A Dataset for Document Visual Question Answering on Multiple Imagescited by 175
  26. 2020Open Question Answering over Tables and Textcited by 163
  27. 2023UniChart: A Universal Vision-language Pretrained Model for Chart Comprehension and Reasoningcited by 158
  28. 2024Multimodal ArXiv: A Dataset for Improving Scientific Comprehension of Large Vision-Language Modelscited by 139
  29. 2022UniGeo: Unifying Geometry Logical Reasoning via Reformulating Mathematical Expressioncited by 137
  30. 2018emrQA: A Large Corpus for Question Answering on Electronic Medical Recordscited by 134
  31. 2024SPHINX-X: Scaling Data and Parameters for a Family of Multi-modal Large Language Modelscited by 131
  32. 2024Prometheus-Vision: Vision-Language Model as a Judge for Fine-Grained Evaluationcited by 123
  33. 2021HiTab: A Hierarchical Table Dataset for Question Answering and Natural Language Generationcited by 117
  34. 2023Evaluating Open-Domain Question Answering in the Era of Large Language Modelscited by 110
  35. 2022Fantastic Questions and Where to Find Them: FairytaleQA -- An Authentic Dataset for Narrative Comprehensioncited by 80
  36. 2021SciCap: Generating Captions for Scientific Figurescited by 76
  37. 2019LEAF-QA: Locate, Encode & Attend for Figure Question Answeringcited by 53
  38. 2021AIT-QA: Question Answering Dataset over Complex Tables in the Airline Industrycited by 44

50 most cited of 59 All 59 (JSON)

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2024: 4; 2025: 39; 2026: 32.

75 papers.

Sort:

By area:

  1. 2025Eagle 2.5: Boosting Long-Context Post-Training for Frontier Vision-Language Modelscited by 63
  2. 2025Transforming Science with Large Language Models: A Survey on AI-assisted Scientific Discovery, Experimentation, Content Generation, and Evaluationcited by 41
  3. 2024VisDoM: Multi-Document QA with Visually Rich Elements Using Multimodal Retrieval-Augmented Generationcited by 40
  4. 2025MicroVQA: A Multimodal Reasoning Benchmark for Microscopy-Based Scientific Researchcited by 30
  5. 2024A Survey on Multimodal Benchmarks: In the Era of Large AI Modelscited by 27
  6. 2025FEABench: Evaluating Language Models on Multiphysics Reasoning Abilitycited by 22
  7. 2025AutoP2C: An LLM-Based Agent Framework for Code Repository Generation from Multimodal Content in Academic Paperscited by 20
  8. 2025Probing the Critical Point (CritPt) of AI Reasoning: a Frontier Physics Research Benchmarkcited by 19
  9. 2025PaperArena: An Evaluation Benchmark for Tool-Augmented Agentic Reasoning on Scientific Literaturecited by 15
  10. 2025MAC: A Live Benchmark for Multimodal Large Language Models in Scientific Understandingcited by 12
  11. 2025What Is That Talk About? A Video-to-Text Summarization Dataset for Scientific Presentationscited by 11
  12. 2025A Survey on Parallel Reasoningcited by 11
38 more
  1. 2025AutoTool: Dynamic Tool Selection and Integration for Agentic Reasoningcited by 11
  2. 2024SCITAT: A Question Answering Benchmark for Scientific Tables and Text Covering Diverse Reasoning Typescited by 10
  3. 2025PosterSum: A Multimodal Benchmark for Scientific Poster Summarizationcited by 10
  4. 2025LLM-based HSE Compliance Assessment: Benchmark, Performance, and Advancementscited by 9
  5. 2025Scaling Beyond Context: A Survey of Multimodal Retrieval-Augmented Generation for Document Understandingcited by 9
  6. 2026What Does Vision Tool-Use Reinforcement Learning Really Learn? Disentangling Tool-Induced and Intrinsic Effects for Crop-and-Zoomcited by 9
  7. 2025MuSciClaims: Multimodal Scientific Claim Verificationcited by 8
  8. 2025DocHop-QA: Towards Multi-Hop Reasoning over Multimodal Document Collectionscited by 8
  9. 2024FM2DS: Few-Shot Multimodal Multihop Data Synthesis with Knowledge Distillation for Question Answeringcited by 7
  10. 2025SciAgent: A Unified Multi-Agent System for Generalistic Scientific Reasoningcited by 6
  11. 2025Table Question Answering in the Era of Large Language Models: A Comprehensive Survey of Tasks, Methods, and Evaluationcited by 5
  12. 2026CiteVQA: Benchmarking Evidence Attribution for Trustworthy Document Intelligencecited by 5
  13. 2025IRLBench: A Multi-modal, Culturally Grounded, Parallel Irish-English Benchmark for Open-Ended LLM Reasoning Evaluationcited by 4
  14. 2025BioMol-MQA: A Multi-Modal Question Answering Dataset For LLM Reasoning Over Bio-Molecular Interactionscited by 4
  15. 2025CEMTM: Contextual Embedding-based Multimodal Topic Modelingcited by 4
  16. 2025CMT-Benchmark: A Benchmark for Condensed Matter Theory Built by Expert Researcherscited by 4
  17. 2025PRISMM-Bench: A Benchmark of Peer-Review Grounded Multimodal Inconsistenciescited by 4
  18. 2026RPC-Bench: A Fine-grained Benchmark for Research Paper Comprehensioncited by 4
  19. 2026TikZilla: Scaling Text-to-TikZ with High-Quality Data and Reinforcement Learningcited by 4
  20. 2026LitTraceQA: A Benchmark for Multi-Stage Grounding and Verification in Scientific Question Answeringcited by 4
  21. 2025T2-RAGBench: Text-and-Table Benchmark for Evaluating Retrieval-Augmented Generationcited by 3
  22. 2025CGBench: Benchmarking Language Model Scientific Reasoning for Clinical Genetics Researchcited by 3
  23. 2025AI Benchmark Democratization and Carpentrycited by 3
  24. 2026BRIDGE: Benchmark for multi-hop Reasoning In long multimodal Documents with Grounded Evidencecited by 3
  25. 2025Doc2SAR: A Synergistic Framework for High-Fidelity Extraction of Structure-Activity Relationships from Scientific Documentscited by 2
  26. 2025LLM-based Question-Answer Framework for Sensor-driven HVAC System Interactioncited by 2
  27. 2025Enhancing Scientific Visual Question Answering through Multimodal Reasoning and Ensemble Modelingcited by 2
  28. 2025GRAFT: GRaPH and Table Reasoning for Textual Alignment -- A Benchmark for Structured Instruction Following and Visual Reasoningcited by 2
  29. 2025An MLCommons Scientific Benchmarks Ontologycited by 2
  30. 2026How to Train Your Long-Context Visual Document Modelcited by 2
  31. 2026IRPAPERS: A Visual Document Benchmark for Scientific Retrieval and Question Answeringcited by 2
  32. 2026PaperTrail: A Claim-Evidence Interface for Grounding Provenance in LLM-based Scholarly Q&Acited by 2
  33. 2026SciMDR: Advancing Scientific Multimodal Document Reasoningcited by 2
  34. 2026EpiBench: Benchmarking Multi-turn Research Workflows for Multimodal Agentscited by 2
  35. 2026MCERF: Advancing Multimodal LLM Evaluation of Engineering Documentation with Enhanced Retrievalcited by 2
  36. 2026Comparative Study of Domain-adapted VLMs for General Document Visual Question Answeringcited by 2
  37. 2025HCT-QA: A Benchmark for Question Answering on Human-Centric Tablescited by 1
  38. 2025Revisiting Model Inversion Evaluation: From Misleading Standards to Reliable Privacy Assessmentcited by 1

50 most cited of 75 All 75 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025SciEGQA: A Dataset for Scientific Evidence-Grounded Question Answering and Reasoning
  2. 2026VinQA: Visual Elements Interleaved Long-form Answer Generation for Real-World Multimodal Document QA
  3. 2023SciGraphQA: A Large-Scale Synthetic Multi-Turn Question-Answering Dataset for Scientific Graphs
  4. 2025A Simple Data Augmentation Strategy for Text-in-Image Scientific VQA
  5. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering
  6. 2026What Lies Beneath: A Call for Distribution-based Visual Question & Answer Datasets
  7. 2026HANIA: Planner-Guided Multimodal Graph Evidence Selection for Grounded Question Answering
  8. 2025Benchmarking Retrieval-Augmented Multimodal Generation for Document Question Answering