mlsanityData to 7 Oct 2026

QA Dataset Explosion: A Taxonomy of NLP Resources for Question Answering and Reading Comprehension

arXiv 2107.12708 ↗First posted 27 Jul 2021cs.CL cs.AIcited by 91

Not on the map. Why?

Abstract (arXiv)

Alongside huge volumes of research on deep learning models in NLP in the recent years, there has been also much work on benchmark datasets needed to track modeling progress. Question answering and reading comprehension have been particularly prolific in this regard, with over 80 new datasets appearing in the past two years. This study is the largest survey of the field to date. We provide an overview of the various formats and domains of the current resources, highlighting the current lacunae for future work. We further discuss the current classifications of "skills" that question answering/reading comprehension systems are supposed to acquire, and propose a new taxonomy. The supplementary materials survey the current multilingual resources and monolingual resources for languages other than English, and we discuss the implications of over-focusing on English. The study is aimed at both practitioners looking for pointers to the wealth of existing data, and at researchers working on new resources.

Key facts. Cited by 91 papers in the map (data through 7 Oct 2026). Builds on 214 of its 348 references that are in the map. First posted on arXiv 27 Jul 2021.

Builds on

214 of 348 references.

References · 214

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2016"Why Should I Trust You?": Explaining the Predictions of Any Classifiercited by 5,040
  5. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  6. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  7. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  8. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  9. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  10. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  11. 2015VQA: Visual Question Answeringcited by 2,489
  12. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
38 more
  1. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
  2. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,793
  3. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,538
  4. 2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,445
  5. 2018Datasheets for Datasetscited by 1,371
  6. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,371
  7. 2018FEVER: a large-scale dataset for Fact Extraction and VERificationcited by 1,362
  8. 2019PubMedQA: A Dataset for Biomedical Research Question Answeringcited by 1,271
  9. 2021BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Modelscited by 1,225
  10. 2018Model Cards for Model Reportingcited by 1,068
  11. 2015Teaching Machines to Read and Comprehendcited by 1,039
  12. 2020Beyond Accuracy: Behavioral Testing of NLP models with CheckListcited by 979
  13. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 960
  14. 2019DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphscited by 883
  15. 2019Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inferencecited by 864
  16. 2017RACE: Large-scale ReAding Comprehension Dataset From Examinationscited by 851
  17. 2017Reading Wikipedia to Answer Open-Domain Questionscited by 777
  18. 2018MultiWOZ -- A Large-Scale Multi-Domain Wizard-of-Oz Dataset for Task-Oriented Dialogue Modellingcited by 760
  19. 2019Universal Adversarial Triggers for Attacking and Analyzing NLPcited by 709
  20. 2017Seq2SQL: Generating Structured Queries from Natural Language using Reinforcement Learningcited by 679
  21. 2018CoQA: A Conversational Question Answering Challengecited by 674
  22. 2018Annotation Artifacts in Natural Language Inference Datacited by 656
  23. 2017The NarrativeQA Reading Comprehension Challengecited by 590
  24. 2019On the Cross-lingual Transferability of Monolingual Representationscited by 575
  25. 2017Adversarial Examples for Evaluating Reading Comprehension Systemscited by 568
  26. 2019ELI5: Long Form Question Answeringcited by 567
  27. 2018Wizard of Wikipedia: Knowledge-Powered Conversational agentscited by 530
  28. 2015Compositional Semantic Parsing on Semi-Structured Tablescited by 509
  29. 2020TyDi QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse Languagescited by 486
  30. 2020UnifiedQA: Crossing Format Boundaries With a Single QA Systemcited by 475
  31. 2021A Dataset of Information-Seeking Questions and Answers Anchored in Research Paperscited by 475
  32. 2018TVQA: Localized, Compositional Video Question Answeringcited by 463
  33. 2018A Corpus for Reasoning About Natural Language Grounded in Photographscited by 457
  34. 2019Learning the Difference that Makes a Difference with Counterfactually-Augmented Datacited by 428
  35. 2017Zero-Shot Relation Extraction via Reading Comprehensioncited by 425
  36. 2018The Web as a Knowledge-base for Answering Complex Questionscited by 409
  37. 2021TAT-QA: A Question Answering Benchmark on a Hybrid of Tabular and Textual Content in Financecited by 402
  38. 2020AmbigQA: Answering Ambiguous Open-domain Questionscited by 385

50 most cited of 214 All 214 (JSON)

Built on by

Cited by year

Citing papers per year: 2021: 14; 2022: 25; 2023: 29; 2024: 13; 2025: 10.

91 papers.

Sort:

By area:

  1. 2023A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivitycited by 888
  2. 2021QuALITY: Question Answering with Long Input Texts, Yes!cited by 204
  3. 2023FLASK: Fine-grained Language Model Evaluation based on Alignment Skill Setscited by 198
  4. 2023Benchmarking Foundation Models with Language-Model-as-an-Examinercited by 172
  5. 2023On the Risk of Misinformation Pollution with Large Language Modelscited by 124
  6. 2023Evaluating Open-Domain Question Answering in the Era of Large Language Modelscited by 110
  7. 2021TopiOCQA: Open-domain Conversational Question Answering with Topic Switchingcited by 95
  8. 2023Graph Neural Prompting with Large Language Modelscited by 64
  9. 2023When Do Annotator Demographics Matter? Measuring the Influence of Annotator Demographics with the POPQUORN Datasetcited by 51
  10. 2025MTRAG: A Multi-Turn Conversational Benchmark for Evaluating Retrieval-Augmented Generation Systemscited by 50
  11. 2022Don't Blame the Annotator: Bias Already Starts in the Annotation Instructionscited by 47
  12. 2023NEWTON: Are Large Language Models Capable of Physical Reasoning?cited by 45
38 more
  1. 2024Reasoning Runtime Behavior of a Program with LLM: How Far Are We?cited by 44
  2. 2023In ChatGPT We Trust? Measuring and Characterizing the Reliability of ChatGPTcited by 35
  3. 2024CaLMQA: Exploring culturally specific long-form question answering across 23 languagescited by 31
  4. 2021Challenges in Generalization in Open Domain Question Answeringcited by 30
  5. 2021ArchivalQA: A Large-scale Benchmark Dataset for Open Domain Question Answering over Historical News Collectionscited by 30
  6. 2023On Efficient Training of Large-Scale Deep Learning Models: A Literature Reviewcited by 26
  7. 2022Beyond Counting Datasets: A Survey of Multilingual Dataset Construction and Necessary Resourcescited by 25
  8. 2024CLAPNQ: Cohesive Long-form Answers from Passages in Natural Questions for RAG systemscited by 23
  9. 2023Position: Key Claims in LLM Research Have a Long Tail of Footnotescited by 20
  10. 2023Make Every Example Count: On the Stability and Utility of Self-Influence for Learning from Noisy NLP Datasetscited by 18
  11. 2024Text Generation: A Systematic Literature Review of Tasks, Evaluation, and Challengescited by 18
  12. 2021MFAQ: a Multilingual FAQ Datasetcited by 16
  13. 2023BHASA: A Holistic Southeast Asian Linguistic and Cultural Evaluation Suite for Large Language Modelscited by 16
  14. 2021MetaQA: Combining Expert Agents for Multi-Skill Question Answeringcited by 15
  15. 2022Low-Resource Dense Retrieval for Open-Domain Question Answering: A Comprehensive Surveycited by 14
  16. 2022Machine Reading, Fast and Slow: When Do Models "Understand" Language?cited by 13
  17. 2022Lila: A Unified Benchmark for Mathematical Reasoningcited by 13
  18. 2023Querying Large Language Models with SQLcited by 13
  19. 2022Table Retrieval May Not Necessitate Table-specific Model Designcited by 11
  20. 2022EduQG: A Multi-format Multiple Choice Dataset for the Educational Domaincited by 11
  21. 2023MAUPQA: Massive Automatically-created Polish Question Answering Datasetcited by 11
  22. 2021A Survey on Legal Question Answering Systemscited by 10
  23. 2022COPA-SSE: Semi-structured Explanations for Commonsense Reasoningcited by 8
  24. 2022ELQA: A Corpus of Metalinguistic Questions and Answers about Englishcited by 8
  25. 2022The Legal Argument Reasoning Task in Civil Procedurecited by 8
  26. 2021Knowledge-Augmented Language Models for Cause-Effect Relation Classificationcited by 7
  27. 2022Mediators: Conversational Agents Explaining NLP Model Behaviorcited by 7
  28. 2023NorQuAD: Norwegian Question Answering Datasetcited by 7
  29. 2024TriviaHG: A Dataset for Automatic Hint Generation from Factoid Questionscited by 7
  30. 2024RAG-based Question Answering over Heterogeneous Data and Textcited by 7
  31. 2022UKP-SQUARE: An Online Platform for Question Answering Researchcited by 6
  32. 2022CC-Riddle: A Question Answering Dataset of Chinese Character Riddlescited by 6
  33. 2022ShortcutLens: A Visual Analytics Approach for Exploring Shortcuts in Natural Language Understanding Datasetcited by 6
  34. 2023One Pass Streaming Algorithm for Super Long Token Attention Approximation in Sublinear Spacecited by 6
  35. 2024FinLLMs: A Framework for Financial Reasoning Dataset Generation with Large Language Modelscited by 6
  36. 2025QuIM-RAG: Advancing Retrieval-Augmented Generation with Inverted Question Matching for Enhanced QA Performancecited by 6
  37. 2021A Survey on non-English Question Answering Datasetcited by 5
  38. 2022The Impact of Cross-Lingual Adjustment of Contextual Word Representations on Zero-Shot Transfercited by 5

50 most cited of 91 All 91 (JSON)