mlsanityData to 7 Oct 2026

Question and Answer Test-Train Overlap in Open-Domain Question Answering Datasets

arXiv 2008.02637 ↗First posted 6 Aug 2020cs.CL cs.AIcited by 137

Not on the map. Why?

Abstract (arXiv)

Ideally Open-Domain Question Answering models should exhibit a number of competencies, ranging from simply memorizing questions seen at training time, to answering novel question formulations with answers seen during training, to generalizing to completely novel questions with novel answers. However, single aggregated test set scores do not show the full picture of what capabilities models truly have. In this work, we perform a detailed study of the test sets of three popular open-domain benchmark datasets with respect to these competencies. We find that 60-70% of test-time answers are also present somewhere in the training sets. We also find that 30% of test-set questions have a near-duplicate paraphrase in their corresponding training sets. Using these findings, we evaluate a variety of popular open-domain models to obtain greater insight into what extent they can actually generalize, and what drives their overall performance. We find that all models perform dramatically worse on questions that cannot be memorized from training sets, with a mean absolute performance difference of 63% between repeated and non-repeated data. Finally we show that simple nearest-neighbor models out-perform a BART closed-book QA model, further highlighting the role that training set memorization plays in these benchmarks

Key facts. Cited by 137 papers in the map (data through 7 Oct 2026). First posted on arXiv 6 Aug 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 50; 2022: 46; 2023: 20; 2024: 10; 2025: 8; 2026: 3.

137 papers.

Sort:

By area:

  1. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,722
  2. 2021Improving language models by retrieving from trillions of tokenscited by 1,278
  3. 2021Ethical and social risks of harm from Language Modelscited by 957
  4. 2021MuSiQue: Multihop Questions via Single-hop Question Compositioncited by 893
  5. 2023REPLUG: Retrieval-Augmented Black-Box Language Modelscited by 786
  6. 2021Dynabench: Rethinking Benchmarking in NLPcited by 447
  7. 2021Entity-Based Knowledge Conflicts in Question Answeringcited by 349
  8. 2021Time-Aware Language Models as Temporal Knowledge Basescited by 290
  9. 2022Teaching language models to support answers with verified quotescited by 290
  10. 2021Mind the Gap: Assessing Temporal Generalization in Neural Language Modelscited by 226
  11. 2021PAQ: 65 Million Probably-Asked Questions and What You Can Do With Themcited by 198
  12. 2021FeTaQA: Free-form Table Question Answeringcited by 174
38 more
  1. 2021Towards Continual Knowledge Learning of Language Modelscited by 172
  2. 2021Hurdles to Progress in Long-form Question Answeringcited by 166
  3. 2022Impact of Pretraining Term Frequencies on Few-Shot Reasoningcited by 142
  4. 2022Rich Knowledge Sources Bring Complex Knowledge Conflicts: Recalibrating Models to Reflect Conflicting Evidencecited by 142
  5. 2021Simple Entity-Centric Questions Challenge Dense Retrieverscited by 140
  6. 2024Benchmark Data Contamination of Large Language Models: A Surveycited by 138
  7. 2022TimeLMs: Diachronic Language Models from Twittercited by 129
  8. 2022Improving the Domain Adaptation of Retrieval Augmented Generation (RAG) Models for Open Domain Question Answeringcited by 126
  9. 2023Speak, Memory: An Archaeology of Books Known to ChatGPT/GPT-4cited by 123
  10. 2022TemporalWiki: A Lifelong Benchmark for Training and Evaluating Ever-Evolving Language Modelscited by 117
  11. 2022Autoregressive Search Engines: Generating Substrings as Document Identifierscited by 112
  12. 2021Case-based Reasoning for Natural Language Queries over Knowledge Basescited by 107
  13. 2021Measure and Improve Robustness in NLP Models: A Surveycited by 107
  14. 2021Question Answering Over Temporal Knowledge Graphscited by 99
  15. 2022State-of-the-art generalisation research in NLP: A taxonomy and reviewcited by 97
  16. 2023CoMPosT: Characterizing and Evaluating Caricature in LLM Simulationscited by 94
  17. 2022DisentQA: Disentangling Parametric and Contextual Knowledge with Counterfactual Question Answeringcited by 93
  18. 2023LoRAMoE: Alleviate World Knowledge Forgetting in Large Language Models via MoE-Style Plugincited by 82
  19. 2021Improving Question Answering Model Robustness with Synthetic Adversarial Data Generationcited by 77
  20. 2022Iteratively Prompt Pre-trained Language Models for Chain of Thoughtcited by 75
  21. 2022CREPE: Open-Domain Question Answering with False Presuppositionscited by 70
  22. 2021Memorization vs. Generalization: Quantifying Data Leakage in NLP Performance Evaluationcited by 69
  23. 2022Recitation-Augmented Language Modelscited by 65
  24. 2024Calibrating Large Language Models Using Their Generations Onlycited by 63
  25. 2021Can Generative Pre-trained Language Models Serve as Knowledge Bases for Closed-book QA?cited by 62
  26. 2021Salient Phrase Aware Dense Retrieval: Can a Dense Retriever Imitate a Sparse One?cited by 59
  27. 2025Can We Trust AI Benchmarks? An Interdisciplinary Review of Current Issues in AI Evaluationcited by 57
  28. 2021GooAQ: Open Question Answering with Diverse Answer Typescited by 53
  29. 2021Dynaboard: An Evaluation-As-A-Service Platform for Holistic Next-Generation Benchmarkingcited by 53
  30. 2024Can Long-Context Language Models Subsume Retrieval, RAG, SQL, and More?cited by 53
  31. 2021NeurIPS 2020 EfficientQA Competition: Systems, Analyses and Lessons Learnedcited by 52
  32. 2021Changing the World by Changing the Datacited by 51
  33. 2025Which of These Best Describes Multiple Choice Evaluation with LLMs? A) Forced B) Flawed C) Fixable D) All of the Abovecited by 51
  34. 2021Learning to Retrieve Passages without Supervisioncited by 46
  35. 2021Multi-task Retrieval for Knowledge-Intensive Taskscited by 44
  36. 2024RAG-QA Arena: Evaluating Domain Robustness for Long-form Retrieval Augmented Question Answeringcited by 43
  37. 2021UnitedQA: A Hybrid Approach for Open Domain Question Answeringcited by 42
  38. 2022QAMPARI: An Open-domain Question Answering Benchmark for Questions with Many Answers from Multiple Paragraphscited by 42

50 most cited of 137 All 137 (JSON)