mlsanityData to 14 Oct 2026

How Much Reading Does Reading Comprehension Require? A Critical Investigation of Popular Benchmarks

arXiv 1808.04926 ↗First posted 14 Aug 2018cs.CL cs.AI cs.LGcited by 88

Not on the map. Why?

Abstract (arXiv)

Many recent papers address reading comprehension, where examples consist of (question, passage, answer) tuples. Presumably, a model must combine information from both questions and passages to predict corresponding answers. However, despite intense interest in the topic, with hundreds of published papers vying for leaderboard dominance, basic questions about the difficulty of many popular benchmarks remain unanswered. In this paper, we establish sensible baselines for the bAbI, SQuAD, CBT, CNN, and Who-did-What datasets, finding that question- and passage-only models often perform surprisingly well. On 14 out of 20 bAbI tasks, passage-only models achieve greater than 50% accuracy, sometimes matching the full model. Interestingly, while CBT provides 20-sentence stories only the last is needed for comparably accurate prediction. By comparison, SQuAD and CNN appear better-constructed.

Key facts. Cited by 88 papers in the map (data through 14 Oct 2026). First posted on arXiv 14 Aug 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 31; 2022: 27; 2023: 14; 2024: 7; 2025: 6; 2026: 3.

88 papers.

Sort:

By area:

  1. 2021MuSiQue: Multihop Questions via Single-hop Question Compositioncited by 916
  2. 2021MultiModalQA: Complex Question Answering over Text, Tables and Imagescited by 179
  3. 2021Hurdles to Progress in Long-form Question Answeringcited by 167
  4. 2022On Feature Learning in the Presence of Spurious Correlationscited by 145
  5. 2022Towards Faithful Model Explanation in NLP: A Surveycited by 130
  6. 2022MultiHiertt: Numerical Reasoning over Multi Hierarchical Tabular and Textual Datacited by 121
  7. 2022State-of-the-art generalisation research in NLP: A taxonomy and reviewcited by 98
  8. 2024Michelangelo: Long Context Evaluations Beyond Haystacks via Latent Structure Queriescited by 88
  9. 2022GLUE-X: Evaluating Natural Language Understanding Models from an Out-of-distribution Generalization Perspectivecited by 72
  10. 2024Artifacts or Abduction: How Do LLMs Answer Multiple-Choice Questions Without the Question?cited by 71
  11. 2022Domain-Adjusted Regression or: ERM May Already Learn Features Sufficient for Out-of-Distribution Generalizationcited by 69
  12. 2022Measuring Causal Effects of Data Statistics on Language Model's `Factual' Predictionscited by 57
38 more
  1. 2024Elements of World Knowledge (EWoK): A Cognition-Inspired Framework for Evaluating Basic World Knowledge in Language Modelscited by 55
  2. 2021Dynaboard: An Evaluation-As-A-Service Platform for Holistic Next-Generation Benchmarkingcited by 53
  3. 2021NeurIPS 2020 EfficientQA Competition: Systems, Analyses and Lessons Learnedcited by 52
  4. 2021Variational Information Bottleneck for Effective Low-Resource Fine-Tuningcited by 51
  5. 2023HAE-RAE Bench: Evaluation of Korean Knowledge in Language Modelscited by 47
  6. 2021Attacking Open-domain Question Answering by Injecting Misinformationcited by 41
  7. 2021COM2SENSE: A Commonsense Reasoning Benchmark with Complementary Sentencescited by 40
  8. 2021Why Machine Reading Comprehension Models Learn Shortcuts?cited by 40
  9. 2022CONDAQA: A Contrastive Reading Comprehension Dataset for Reasoning about Negationcited by 39
  10. 2021Back to Square One: Artifact Detection, Training and Commonsense Disentanglement in the Winograd Schemacited by 32
  11. 2021On the Efficacy of Adversarial Data Collection for Question Answering: Results from a Large-Scale Randomized Studycited by 31
  12. 2021Challenges in Generalization in Open Domain Question Answeringcited by 30
  13. 2022DISCO: Distilling Counterfactuals with Large Language Modelscited by 30
  14. 2021Sensitivity as a Complexity Measure for Sequence Classification Taskscited by 28
  15. 2023It Takes Two to Tango: Navigating Conceptualizations of NLP Tasks and Measurements of Performancecited by 26
  16. 2023Explanation-based Finetuning Makes Models More Robust to Spurious Cuescited by 25
  17. 2021What Ingredients Make for an Effective Crowdsourcing Protocol for Difficult NLU Data Collection Tasks?cited by 23
  18. 2021Avoiding Inference Heuristics in Few-shot Prompt-based Finetuningcited by 23
  19. 2023What's the Meaning of Superhuman Performance in Today's NLU?cited by 23
  20. 2021A Few More Examples May Be Worth Billions of Parameterscited by 19
  21. 2021On the Robustness of Reading Comprehension Models to Entity Renamingcited by 17
  22. 2022The worst of both worlds: A comparative analysis of errors in learning from data in psychology and machine learningcited by 17
  23. 2021Uncertainty Calibration for Ensemble-Based Debiasing Methodscited by 16
  24. 2022Does Your Model Classify Entities Reasonably? Diagnosing and Mitigating Spurious Correlations in Entity Typingcited by 16
  25. 2021NOAHQA: Numerical Reasoning with Interpretable Graph Question Answering Datasetcited by 14
  26. 2022Less Learn Shortcut: Analyzing and Mitigating Learning of Spurious Feature-Label Correlationcited by 13
  27. 2022Machine Reading, Fast and Slow: When Do Models "Understand" Language?cited by 13
  28. 2021ParaShoot: A Hebrew Question Answering Datasetcited by 12
  29. 2021Exploring Strategies for Generalizable Commonsense Reasoning with Pre-trained Modelscited by 11
  30. 2022What Makes Reading Comprehension Questions Difficult?cited by 11
  31. 2022Feature-Level Debiased Natural Language Understandingcited by 10
  32. 2023Multimodal Graph Transformer for Multimodal Question Answeringcited by 10
  33. 2025Do Activation Verbalization Methods Convey Privileged Information?cited by 10
  34. 2021Text-based NP Enrichmentcited by 8
  35. 2021Dyna-bAbI: unlocking bAbI's potential with dynamic synthetic benchmarkingcited by 7
  36. 2024Desiderata for the Context Use of Question Answering Systemscited by 7
  37. 2021On the Interplay Between Fine-tuning and Composition in Transformerscited by 6
  38. 2022Partial-input baselines show that NLI models can ignore context, but they don'tcited by 6

50 most cited of 88 All 88 (JSON)