# SciDQA: A Deep Reading Comprehension Dataset over Scientific Papers

arXiv: https://arxiv.org/abs/2411.05338 · HTML: https://mlsanity.com/paper/2411.05338/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 46 references that are in the map.
- First posted on arXiv 8 Nov 2024.

## Builds on (most cited first, 21 of 21)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [PubMedQA: A Dataset for Biomedical Research Question Answering](https://mlsanity.com/paper/1909.06146/) (2019)
- [DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphs](https://mlsanity.com/paper/1903.00161/) (2019)
- [Galactica: A Large Language Model for Science](https://mlsanity.com/paper/2211.09085/) (2022)
- [How Much Knowledge Can You Pack Into the Parameters of a Language Model?](https://mlsanity.com/paper/2002.08910/) (2020)
- [The Falcon Series of Open Language Models](https://mlsanity.com/paper/2311.16867/) (2023)
- [A Dataset of Information-Seeking Questions and Answers Anchored in Research Papers](https://mlsanity.com/paper/2105.03011/) (2021)
- [Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Models](https://mlsanity.com/paper/2405.01535/) (2024)
- [Nougat: Neural Optical Understanding for Academic Documents](https://mlsanity.com/paper/2308.13418/) (2023)
- [Revisiting the Gold Standard: Grounding Summarization Evaluation with Robust Human Evaluation](https://mlsanity.com/paper/2212.07981/) (2022)
- [A Dataset of Peer Reviews (PeerRead): Collection, Insights and NLP Applications](https://mlsanity.com/paper/1804.09635/) (2018)
- [Learning Compact Metrics for MT](https://mlsanity.com/paper/2110.06341/) (2021)
- [BIOMRC: A Dataset for Biomedical Machine Reading Comprehension](https://mlsanity.com/paper/2005.06376/) (2020)
- [ARIES: A Corpus of Scientific Paper Edits Made in Response to Peer Reviews](https://mlsanity.com/paper/2306.12587/) (2023)
- [ReIFE: Re-evaluating Instruction-Following Evaluation](https://mlsanity.com/paper/2410.07069/) (2024)
- [Relatedly: Scaffolding Literature Reviews with Existing Related Work Sections](https://mlsanity.com/paper/2302.06754/) (2023)
- [The Semantic Reader Project: Augmenting Scholarly Documents through AI-Powered Interactive Reading Interfaces](https://mlsanity.com/paper/2303.14334/) (2023)

## Built on by (most cited first, 15 of 15)

- [Survey on Evaluation of LLM-based Agents](https://mlsanity.com/paper/2503.16416/) (2025)
- [ResearchQA: Evaluating Scholarly Question Answering at Scale Across 75 Fields with Survey-Mined Questions and Rubrics](https://mlsanity.com/paper/2509.00496/) (2025)
- [PeerQA: A Scientific Question Answering Dataset from Peer Reviews](https://mlsanity.com/paper/2502.13668/) (2025)
- [SciArena: An Open Evaluation Platform for Non-Verifiable Scientific Literature-Grounded Tasks](https://mlsanity.com/paper/2507.01001/) (2025)
- [Evolving Roles of LLMs in Scientific Innovation: Assistant, Collaborator, Scientist, and Evaluator](https://mlsanity.com/paper/2507.11810/) (2025)
- [Generative AI in Transportation Planning: A Survey](https://mlsanity.com/paper/2503.07158/) (2025)
- [MuSciClaims: Multimodal Scientific Claim Verification](https://mlsanity.com/paper/2506.04585/) (2025)
- [SciRAG: Adaptive, Citation-Aware, and Outline-Guided Retrieval and Synthesis for Scientific Literature](https://mlsanity.com/paper/2511.14362/) (2025)
- [NeuSym-RAG: Hybrid Neural Symbolic Retrieval with Multiview Structuring for PDF Question Answering](https://mlsanity.com/paper/2505.19754/) (2025)
- [SciMDR: Advancing Scientific Multimodal Document Reasoning](https://mlsanity.com/paper/2603.12249/) (2026)
- [IntelliAsk: Learning to Ask High-Quality Research Questions via RLVR](https://mlsanity.com/paper/2602.15849/) (2026)
- [CareMedEval dataset: Evaluating Critical Appraisal and Reasoning in the Biomedical Field](https://mlsanity.com/paper/2511.03441/) (2025)
- [PaperGuide: Making Small Language-Model Paper-Reading Agents More Efficient](https://mlsanity.com/paper/2601.12988/) (2026)
- [LECTOR: Joint Optimization of Scientific Reasoning Graphs and Introduction Generation](https://mlsanity.com/paper/2605.25964/) (2026)
- [Metag: A dataset to build agentic meta-reviewing capabilities](https://mlsanity.com/paper/2608.20488/) (2026)

## Abstract (arXiv)

Scientific literature is typically dense, requiring significant background knowledge and deep comprehension for effective engagement. We introduce SciDQA, a new dataset for reading comprehension that challenges LLMs for a deep understanding of scientific articles, consisting of 2,937 QA pairs. Unlike other scientific QA datasets, SciDQA sources questions from peer reviews by domain experts and answers by paper authors, ensuring a thorough examination of the literature. We enhance the dataset's quality through a process that carefully filters out lower quality questions, decontextualizes the content, tracks the source document across different versions, and incorporates a bibliography for multi-document question-answering. Questions in SciDQA necessitate reasoning across figures, tables, equations, appendices, and supplementary materials, and require multi-document reasoning. We evaluate several open-source and proprietary LLMs across various configurations to explore their capabilities in generating relevant and factual responses. Our comprehensive evaluation, based on metrics for surface-level similarity and LLM judgements, highlights notable performance discrepancies. SciDQA represents a rigorously curated, naturally derived scientific QA dataset, designed to facilitate research on complex scientific text understanding.