mlsanityData to 7 Oct 2026

A Dataset of Information-Seeking Questions and Answers Anchored in Research Papers

arXiv 2105.03011 ↗First posted 7 May 2021cs.CLcited by 475

Not on the map. Why?

Cited by 66 in Retrieval-augmented generation, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Readers of academic research papers often read with the goal of answering specific questions. Question Answering systems that can answer those questions can make consumption of the content much more efficient. However, building such tools requires data that reflect the difficulty of the task arising from complex reasoning about claims made in multiple parts of a paper. In contrast, existing information-seeking question answering datasets usually contain questions about generic factoid-type information. We therefore present QASPER, a dataset of 5,049 questions over 1,585 Natural Language Processing papers. Each question is written by an NLP practitioner who read only the title and abstract of the corresponding paper, and the question seeks information present in the full text. The questions are then answered by a separate set of NLP practitioners who also provide supporting evidence to answers. We find that existing models that do well on other QA tasks do not perform well on answering these questions, underperforming humans by at least 27 F1 points when answering them from entire papers, motivating further research in document-grounded, information-seeking QA, which our dataset is designed to facilitate.

Key facts. Cited by 475 papers in the map (data through 7 Oct 2026). Builds on 30 of its 40 references that are in the map. First posted on arXiv 7 May 2021.

Builds on

30 of 40 references.

References · 30

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  6. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  7. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  8. 2020Longformer: The Long-Document Transformercited by 3,362
  9. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  10. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,126
  11. 2016MS MARCO: A Human Generated MAchine Reading COmprehension Datasetcited by 1,591
  12. 2019PubMedQA: A Dataset for Biomedical Research Question Answeringcited by 1,271
18 more
  1. 2019DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphscited by 883
  2. 2016Training Deep Nets with Sublinear Memory Costcited by 825
  3. 2018CoQA: A Conversational Question Answering Challengecited by 674
  4. 2017The NarrativeQA Reading Comprehension Challengecited by 590
  5. 2020TyDi QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse Languagescited by 486
  6. 2020UnifiedQA: Crossing Format Boundaries With a Single QA Systemcited by 475
  7. 2018QuAC : Question Answering in Contextcited by 379
  8. 2016NewsQA: A Machine Comprehension Datasetcited by 330
  9. 2017Constructing Datasets for Multi-hop Reading Comprehension Across Documentscited by 190
  10. 2019Learning to Retrieve Reasoning Paths over Wikipedia Graph for Question Answeringcited by 147
  11. 2018emrQA: A Large Corpus for Question Answering on Electronic Medical Recordscited by 134
  12. 2019Quoref: A Reading Comprehension Dataset with Questions Requiring Coreferential Reasoningcited by 100
  13. 2019Question Answering for Privacy Policies: Combining Computational and Legal Perspectivescited by 65
  14. 2020IIRC: A Dataset of Incomplete Information Reading Comprehension Questionscited by 52
  15. 2020BIOMRC: A Dataset for Biomedical Machine Reading Comprehensioncited by 35
  16. 2020PolicyQA: A Reading Comprehension Dataset for Privacy Policiescited by 31
  17. 2020RikiNet: Reading Wikipedia Pages for Natural Question Answeringcited by 21
  18. 2020Easy, Reproducible and Quality-Controlled Data Collection with Crowdaqcited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 7; 2022: 24; 2023: 44; 2024: 128; 2025: 116; 2026: 156.

475 papers.

Sort:

By area:

  1. 2023Retrieval-Augmented Generation for Large Language Models: A Surveycited by 2,185
  2. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  3. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,378
  4. 2024SnapKV: LLM Knows What You are Looking for Before Generationcited by 848
  5. 2022Galactica: A Large Language Model for Sciencecited by 736
  6. 2024RAPTOR: Recursive Abstractive Processing for Tree-Organized Retrievalcited by 519
  7. 2024Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inferencecited by 487
  8. 2024Gated Delta Networks: Improving Mamba2 with Delta Rulecited by 483
  9. 2024PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funnelingcited by 420
  10. 2024LongBench v2: Towards Deeper Understanding and Reasoning on Realistic Long-context Multitaskscited by 367
  11. 2024∞Bench: Extending Long Context Evaluation Beyond 100K Tokenscited by 360
  12. 2023Effective Long-Context Scaling of Foundation Modelscited by 301
38 more
  1. 2022UL2: Unifying Language Learning Paradigmscited by 292
  2. 2024Long-context LLMs Struggle with Long In-context Learningcited by 285
  3. 2024Nomic Embed: Training a Reproducible Long Context Text Embeddercited by 255
  4. 2025Agentic Retrieval-Augmented Generation: A Survey on Agentic RAGcited by 242
  5. 2025MemAgent: Reshaping Long-Context LLM with Multi-Conv RL-based Memory Agentcited by 235
  6. 2023L-Eval: Instituting Standardized Evaluation for Long Context Language Modelscited by 232
  7. 2024Ada-KV: Optimizing KV Cache Eviction by Adaptive Budget Allocation for Efficient LLM Inferencecited by 230
  8. 2023Camels in a Changing Climate: Enhancing LM Adaptation with Tulu 2cited by 227
  9. 2024Chain of Agents: Large Language Models Collaborating on Long-Context Taskscited by 201
  10. 2025Survey on Evaluation of LLM-based Agentscited by 190
  11. 2022Branch-Train-Merge: Embarrassingly Parallel Training of Expert Language Modelscited by 187
  12. 2024An Empirical Study of Mamba-based Language Modelscited by 177
  13. 2023ZeroSCROLLS: A Zero-Shot Benchmark for Long Text Understandingcited by 175
  14. 2022SCROLLS: Standardized CompaRison Over Long Language Sequencescited by 170
  15. 2025Do LLMs Recognize Your Preferences? Evaluating Personalized Preference Following in LLMscited by 142
  16. 2023Focused Transformer: Contrastive Training for Context Scalingcited by 133
  17. 2021DEMix Layers: Disentangling Domains for Modular Language Modelingcited by 130
  18. 2025AbstentionBench: Reasoning LLMs Fail on Unanswerable Questionscited by 129
  19. 2023Unlimiformer: Long-Range Transformers with Unlimited Length Inputcited by 127
  20. 2024A Survey on Large Language Model Acceleration based on KV Cache Managementcited by 125
  21. 2025Deep Research Agents: A Systematic Examination And Roadmapcited by 122
  22. 2023LM-Infinite: Zero-Shot Extreme Length Generalization for Large Language Modelscited by 120
  23. 2024Transformers are Multi-State RNNscited by 120
  24. 2024Make Your LLM Fully Utilize the Contextcited by 118
  25. 2024MemoRAG: Boosting Long Context Processing with Global Memory-Enhanced Retrieval Augmentationcited by 118
  26. 2024Retrieval Augmented Generation or Long-Context LLMs? A Comprehensive Study and Hybrid Approachcited by 117
  27. 2023AutoMix: Automatically Mixing Language Modelscited by 116
  28. 2024Know Your Limits: A Survey of Abstention in Large Language Modelscited by 116
  29. 2023Retrieval meets Long Context Large Language Modelscited by 114
  30. 2024MagicPIG: LSH Sampling for Efficient LLM Generationcited by 112
  31. 2024LongRAG: Enhancing Retrieval-Augmented Generation with Long-context LLMscited by 104
  32. 2021QA Dataset Explosion: A Taxonomy of NLP Resources for Question Answering and Reading Comprehensioncited by 91
  33. 2023CoLT5: Faster Long-Range Transformers with Conditional Computationcited by 84
  34. 2024Long-Context Language Modeling with Parallel Context Encodingcited by 84
  35. 2024A Simple and Effective L2 Norm-Based Strategy for KV Cache Compressioncited by 84
  36. 2024SPIQA: A Dataset for Multimodal Question Answering on Scientific Paperscited by 75
  37. 2025Cartridges: Lightweight and general-purpose long context representations via self-studycited by 74
  38. 2024Gated Slot Attention for Efficient Linear-Time Sequence Modelingcited by 73

50 most cited of 475 All 475 (JSON)