mlsanityData to 14 Oct 2026

What do Models Learn from Question Answering Datasets?

arXiv 2004.03490 ↗First posted 7 Apr 2020cs.CLcited by 45

Not on the map. Why?

Abstract (arXiv)

While models have reached superhuman performance on popular question answering (QA) datasets such as SQuAD, they have yet to outperform humans on the task of question answering itself. In this paper, we investigate if models are learning reading comprehension from QA datasets by evaluating BERT-based models across five datasets. We evaluate models on their generalizability to out-of-domain examples, responses to missing or incorrect data, and ability to handle question variations. We find that no single dataset is robust to all of our experiments and identify shortcomings in both datasets and evaluation methods. Following our analysis, we make recommendations for building future QA datasets that better evaluate the task of question answering through reading comprehension. We also release code to convert QA datasets to a shared format for easier experimentation at https://github.com/amazon-research/qa-dataset-converter.

Key facts. Cited by 45 papers in the map (data through 14 Oct 2026). First posted on arXiv 7 Apr 2020.

Builds on

No reference list yet.

Built on by

Cited by year

Citing papers per year: 2021: 14; 2022: 15; 2023: 8; 2024: 6; 2025: 2.

45 papers.

Sort:

  1. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,879
  2. 2021Understanding Dataset Difficulty with V-Usable Informationcited by 291
  3. 2021KLUE: Korean Language Understanding Evaluationcited by 114
  4. 2022Shortcut Learning of Large Language Models in Natural Language Understandingcited by 85
  5. 2021SpartQA: : A Textual Question Answering Benchmark for Spatial Reasoningcited by 77
  6. 2022CONDAQA: A Contrastive Reading Comprehension Dataset for Reasoning about Negationcited by 39
  7. 2022CLIP-TD: CLIP Targeted Distillation for Vision-Language Taskscited by 32
  8. 2022Multimodal Adaptive Distillation for Leveraging Unimodal Encoders for Vision-Language Taskscited by 23
  9. 2022Exploring The Landscape of Distributional Robustness for Question Answering Modelscited by 23
  10. 2021On the Robustness of Reading Comprehension Models to Entity Renamingcited by 17
  11. 2022Machine Reading, Fast and Slow: When Do Models "Understand" Language?cited by 13
  12. 2021Exploring Strategies for Generalizable Commonsense Reasoning with Pre-trained Modelscited by 11
33 more
  1. 2021Can Question Generation Debias Question Answering Models? A Case Study on Question-Context Lexical Overlapcited by 8
  2. 2024Shortcut Learning in In-Context Learning: A Surveycited by 8
  3. 2024Desiderata for the Context Use of Question Answering Systemscited by 7
  4. 2024From Form(s) to Meaning: Probing the Semantic Depths of Language Models Using Multisense Consistencycited by 7
  5. 2021Are Multilingual BERT models robust? A Case Study on Adversarial Attacks for Multilingual Question Answeringcited by 6
  6. 2023Does Deep Learning Learn to Abstract? A Systematic Probing Frameworkcited by 6
  7. 2023Learning to Generalize for Cross-domain QAcited by 6
  8. 2021Toward Deconfounding the Influence of Entity Demographics for Question Answering Accuracycited by 5
  9. 2023Whispers of Doubt Amidst Echoes of Triumph in NLP Robustnesscited by 5
  10. 2024TIGQA:An Expert Annotated Question Answering Dataset in Tigrinyacited by 5
  11. 2022An Empirical Investigation of Commonsense Self-Supervision with Knowledge Graphscited by 4
  12. 2022A Survey on Measuring and Mitigating Reasoning Shortcuts in Machine Reading Comprehensioncited by 4
  13. 2022IDK-MRC: Unanswerable Questions for Indonesian Machine Reading Comprehensioncited by 4
  14. 2023The Impacts of Unanswerable Questions on the Robustness of Machine Reading Comprehension Modelscited by 4
  15. 2024It Is Not About What You Say, It Is About How You Say It: A Surprisingly Simple Approach for Improving Reading Comprehensioncited by 4
  16. 2021Bridging the Gap between Language Model and Reading Comprehension: Unsupervised MRC via Self-Supervisioncited by 3
  17. 2022Trajectory Test-Train Overlap in Next-Location Prediction Datasetscited by 3
  18. 2021Ensemble Learning-Based Approach for Improving Generalization Capability of Machine Reading Comprehension Systemscited by 2
  19. 2021Interactive Machine Comprehension with Dynamic Knowledge Graphscited by 2
  20. 2021Can Edge Probing Tasks Reveal Linguistic Knowledge in QA Models?cited by 2
  21. 2022Extending the Scope of Out-of-Domain: Examining QA models in multiple subdomainscited by 2
  22. 2022Antipatterns in Software Classification Taxonomiescited by 1
  23. 2023Benchmarks for Pir'a 2.0, a Reading Comprehension Dataset about the Ocean, the Brazilian Coast, and Climate Changecited by 1
  24. 2023Implications of Annotation Artifacts in Edge Probing Test Datasetscited by 1
  25. 2024Assessing biomedical knowledge robustness in large language models by query-efficient sampling attackscited by 1
  26. 2025A Survey of the State-of-the-Art in Conversational Question Answering Systemscited by 1
  27. 2021True or False: Does the Deep Learning Model Learn to Detect Rumors?cited by 0
  28. 2021How to Build Robust FAQ Chatbot with Controllable Question Generator?cited by 0
  29. 2022Using calibrator to improve robustness in Machine Reading Comprehensioncited by 0
  30. 2022MRCLens: an MRC Dataset Bias Detection Toolkitcited by 0
  31. 2022Penalizing Confident Predictions on Largely Perturbed Inputs Does Not Improve Out-of-Distribution Generalization in Question Answeringcited by 0
  32. 2023AGent: A Novel Pipeline for Automatically Creating Unanswerable Questionscited by 0
  33. 2025Comparative Analysis of 47 Context-Based Question Answer Models Across 8 Diverse Datasetscited by 0