mlsanityData to 7 Oct 2026

M3SciQA: A Multi-Modal Multi-Document Scientific QA Benchmark for Evaluating Foundation Models

arXiv 2411.04075 ↗First posted 6 Nov 2024cs.CL cs.AIcited by 26

Not on the map. Why?

Abstract (arXiv)

Existing benchmarks for evaluating foundation models mainly focus on single-document, text-only tasks. However, they often fail to fully capture the complexity of research workflows, which typically involve interpreting non-textual data and gathering information across multiple documents. To address this gap, we introduce M3SciQA, a multi-modal, multi-document scientific question answering benchmark designed for a more comprehensive evaluation of foundation models. M3SciQA consists of 1,452 expert-annotated questions spanning 70 natural language processing paper clusters, where each cluster represents a primary paper along with all its cited documents, mirroring the workflow of comprehending a single paper by requiring multi-modal and multi-document data. With M3SciQA, we conduct a comprehensive evaluation of 18 foundation models. Our results indicate that current foundation models still significantly underperform compared to human experts in multi-modal information retrieval and in reasoning across multiple scientific documents. Additionally, we explore the implications of these findings for the future advancement of applying foundation models in multi-modal scientific literature analysis.

Key facts. Cited by 26 papers in the map (data through 7 Oct 2026). Builds on 31 of its 46 references that are in the map. First posted on arXiv 6 Nov 2024.

Builds on

31 of 46 references.

References · 31

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  3. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  4. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  5. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  6. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
  7. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
  8. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,105
  9. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  10. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  11. 2016MS MARCO: A Human Generated MAchine Reading COmprehension Datasetcited by 1,591
  12. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,582
19 more
  1. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,449
  2. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,221
  3. 2021Unsupervised Dense Information Retrieval with Contrastive Learningcited by 1,203
  4. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehensioncited by 929
  5. 2019DialoGPT: Large-Scale Generative Pre-training for Conversational Response Generationcited by 842
  6. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 738
  7. 2024DeepSeek-VL: Towards Real-World Vision-Language Understandingcited by 718
  8. 2021A Dataset of Information-Seeking Questions and Answers Anchored in Research Paperscited by 475
  9. 2022News Summarization and Evaluation in the Era of GPT-3cited by 324
  10. 2024CharXiv: Charting Gaps in Realistic Chart Understanding in Multimodal LLMscited by 229
  11. 2021GeoQA: A Geometric Question Answering Benchmark Towards Multimodal Numerical Reasoningcited by 224
  12. 2021MultiModalQA: Complex Question Answering over Text, Tables and Imagescited by 177
  13. 2018RecipeQA: A Challenge Dataset for Multimodal Comprehension of Cooking Recipescited by 74
  14. 2023Benchmarking Generation and Evaluation Capabilities of Large Language Models for Instruction Controllable Summarizationcited by 69
  15. 2023DocMath-Eval: Evaluating Math Reasoning Capabilities of LLMs in Understanding Long and Specialized Documentscited by 45
  16. 2023Towards Interpretable and Efficient Automatic Reference-Based Summarization Evaluationcited by 24
  17. 2023Information Value: Measuring Utterance Predictability as Distance from Plausible Alternativescited by 22
  18. 2023A Question Answering Framework for Decontextualizing User-facing Snippets from Scientific Documentscited by 15
  19. 2024KIWI: A Dataset of Knowledge-Intensive Writing Instructions for Answering Research Questionscited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 13; 2026: 13.

26 papers.

Sort:

By area:

  1. 2025EmoBench-M: Benchmarking Emotional Intelligence for Multimodal Large Language Modelscited by 32
  2. 2025Large Language Models Meet Knowledge Graphs for Question Answering: Synthesis and Opportunitiescited by 30
  3. 2025SciArena: An Open Evaluation Platform for Non-Verifiable Scientific Literature-Grounded Taskscited by 17
  4. 2025SciVer: Evaluating Foundation Models for Multimodal Scientific Claim Verificationcited by 16
  5. 2025AbGen: Evaluating Large Language Models in Ablation Study Design and Evaluation for Scientific Researchcited by 12
  6. 2025Can Multimodal Foundation Models Understand Schematic Diagrams? An Empirical Study on Information-Seeking QA over Scientific Paperscited by 9
  7. 2025Evolving Roles of LLMs in Scientific Innovation: Assistant, Collaborator, Scientist, and Evaluatorcited by 9
  8. 2025SemBench: A Benchmark for Semantic Query Processing Enginescited by 8
  9. 2025DocHop-QA: Towards Multi-Hop Reasoning over Multimodal Document Collectionscited by 7
  10. 2026LitTraceQA: A Benchmark for Multi-Stage Grounding and Verification in Scientific Question Answeringcited by 4
  11. 2025NeuSym-RAG: Hybrid Neural Symbolic Retrieval with Multiview Structuring for PDF Question Answeringcited by 3
  12. 2025HiSciBench: A Hierarchical Multi-disciplinary Benchmark for Scientific Intelligence from Reading to Discoverycited by 3
14 more
  1. 2025GRAFT: GRaPH and Table Reasoning for Textual Alignment -- A Benchmark for Structured Instruction Following and Visual Reasoningcited by 2
  2. 2026PaperTrail: A Claim-Evidence Interface for Grounding Provenance in LLM-based Scholarly Q&Acited by 2
  3. 2026SciMDR: Advancing Scientific Multimodal Document Reasoningcited by 2
  4. 2025AirQA: A Comprehensive QA Dataset for AI Research with Instance-Level Evaluationcited by 1
  5. 2026An Expert Schema for Evaluating Large Language Model Errors in Scholarly Question-Answering Systemscited by 1
  6. 2026Automatic Inter-document Multi-hop Scientific QA Generationcited by 1
  7. 2026PaperScope: A Multi-Modal Multi-Document Benchmark for Agentic Deep Research Across Massive Scientific Paperscited by 1
  8. 2026Querying Multimodal Scientific Papers with AI: Practices and Preferences Across Blind, Low-Vision, and Sighted Scientistscited by 1
  9. 2026Diagnosing Structural Failures in LLM-Based Evidence Extraction for Meta-Analysiscited by 0
  10. 2026MULTISEISMO: A Multimodal Seismic Dataset and Model for Cross-Modal Seismic Understandingcited by 0
  11. 2026SciR: A Controllable Benchmark for Scientific Reasoning in LLMscited by 0
  12. 2026DocNavRAG: Document-Structured Graph RAG with Stateful Evidence Construction for Complex Document Question Answeringcited by 0
  13. 2026SciDocBench: A Workflow-Centered Benchmark and Data Pipeline for Scientific Document Understandingcited by 0
  14. 2026Amnesia by Design, Memory By Necessity: Persistent State for Document Intelligencecited by 0