mlsanityData to 7 Oct 2026

To Test Machine Comprehension, Start by Defining Comprehension

arXiv 2005.01525 ↗First posted 4 May 2020cs.CL cs.AIcited by 31

Not on the map. Why?

Abstract (arXiv)

Many tasks aim to measure machine reading comprehension (MRC), often focusing on question types presumed to be difficult. Rarely, however, do task designers start by considering what systems should in fact comprehend. In this paper we make two key contributions. First, we argue that existing approaches do not adequately define comprehension; they are too unsystematic about what content is tested. Second, we present a detailed definition of comprehension -- a "Template of Understanding" -- for a widely useful class of texts, namely short narratives. We then conduct an experiment that strongly suggests existing systems are not up to the task of narrative understanding as we define it.

Key facts. Cited by 31 papers in the map (data through 7 Oct 2026). First posted on arXiv 4 May 2020.

Builds on

No reference list yet.

Built on by

Cited by year

Citing papers per year: 2021: 11; 2022: 12; 2023: 4; 2024: 4.

31 papers.

Sort:

By area:

  1. 2021QA Dataset Explosion: A Taxonomy of NLP Resources for Question Answering and Reading Comprehensioncited by 91
  2. 2022Validating Large Language Models with ReLMcited by 33
  3. 2021TellMeWhy: A Dataset for Answering Why-Questions in Narrativescited by 27
  4. 2024Is It Really Long Context if All You Need Is Retrieval? Towards Genuinely Difficult Long Context NLPcited by 27
  5. 2021English Machine Reading Comprehension Datasets: A Surveycited by 26
  6. 2022The Causal News Corpus: Annotating Causal Relations in Event Sentences from Newscited by 24
  7. 2023What's the Meaning of Superhuman Performance in Today's NLU?cited by 22
  8. 2021Discourse Comprehension: A Question Answering Framework to Represent Sentence Connectionscited by 19
  9. 2022Machine Reading, Fast and Slow: When Do Models "Understand" Language?cited by 13
  10. 2022A Survey of Machine Narrative Reading Comprehension Assessmentscited by 10
  11. 2022UniCausal: Unified Benchmark and Repository for Causal Text Miningcited by 9
  12. 2024Natural Language Processing RELIES on Linguisticscited by 9
19 more
  1. 2021Text-based NP Enrichmentcited by 8
  2. 2021Towards General Natural Language Understanding with Probabilistic Worldbuildingcited by 7
  3. 2021Comprehension Based Question Answering using Bloom's Taxonomycited by 7
  4. 2021Dyna-bAbI: unlocking bAbI's potential with dynamic synthetic benchmarkingcited by 7
  5. 2021Improving Question Answering Performance Using Knowledge Distillation and Active Learningcited by 6
  6. 2022A Kind Introduction to Lexical and Grammatical Aspect, with a Survey of Computational Approachescited by 5
  7. 2022POQue: Asking Participant-specific Outcome Questions for a Deeper Understanding of Complex Eventscited by 5
  8. 2024Evaluating LLMs for Targeted Concept Simplification for Domain-Specific Textscited by 5
  9. 2022Hidden Schema Networkscited by 4
  10. 2022A Survey on Measuring and Mitigating Reasoning Shortcuts in Machine Reading Comprehensioncited by 4
  11. 2022Event Causality Identification with Causal News Corpus -- Shared Task 3, CASE 2022cited by 3
  12. 2021Situation and Behavior Understanding by Trope Detection on Filmscited by 2
  13. 2023Reimagining Retrieval Augmented Language Models for Answering Queriescited by 2
  14. 2024Is ChatGPT the Future of Causal Text Mining? A Comprehensive Evaluation and Analysiscited by 2
  15. 2021CogME: A Cognition-Inspired Multi-Dimensional Evaluation Metric for Story Understandingcited by 1
  16. 2022IDIAPers @ Causal News Corpus 2022: Extracting Cause-Effect-Signal Triplets via Pre-trained Autoregressive Language Modelcited by 1
  17. 2023On Degrees of Freedom in Defining and Testing Natural Language Understandingcited by 1
  18. 2022Design considerations for a hierarchical semantic compositional framework for medical natural language understandingcited by 0
  19. 2023On General Language Understandingcited by 0