# To Test Machine Comprehension, Start by Defining Comprehension

arXiv: https://arxiv.org/abs/2005.01525 · HTML: https://mlsanity.com/paper/2005.01525/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 4 May 2020.

## Built on by (most cited first, 30 of 31)

- [QA Dataset Explosion: A Taxonomy of NLP Resources for Question Answering and Reading Comprehension](https://mlsanity.com/paper/2107.12708/) (2021)
- [Validating Large Language Models with ReLM](https://mlsanity.com/paper/2211.15458/) (2022)
- [TellMeWhy: A Dataset for Answering Why-Questions in Narratives](https://mlsanity.com/paper/2106.06132/) (2021)
- [Is It Really Long Context if All You Need Is Retrieval? Towards Genuinely Difficult Long Context NLP](https://mlsanity.com/paper/2407.00402/) (2024)
- [English Machine Reading Comprehension Datasets: A Survey](https://mlsanity.com/paper/2101.10421/) (2021)
- [The Causal News Corpus: Annotating Causal Relations in Event Sentences from News](https://mlsanity.com/paper/2204.11714/) (2022)
- [What's the Meaning of Superhuman Performance in Today's NLU?](https://mlsanity.com/paper/2305.08414/) (2023)
- [Discourse Comprehension: A Question Answering Framework to Represent Sentence Connections](https://mlsanity.com/paper/2111.00701/) (2021)
- [Machine Reading, Fast and Slow: When Do Models "Understand" Language?](https://mlsanity.com/paper/2209.07430/) (2022)
- [A Survey of Machine Narrative Reading Comprehension Assessments](https://mlsanity.com/paper/2205.00299/) (2022)
- [UniCausal: Unified Benchmark and Repository for Causal Text Mining](https://mlsanity.com/paper/2208.09163/) (2022)
- [Natural Language Processing RELIES on Linguistics](https://mlsanity.com/paper/2405.05966/) (2024)
- [Text-based NP Enrichment](https://mlsanity.com/paper/2109.12085/) (2021)
- [Towards General Natural Language Understanding with Probabilistic Worldbuilding](https://mlsanity.com/paper/2105.02486/) (2021)
- [Comprehension Based Question Answering using Bloom's Taxonomy](https://mlsanity.com/paper/2106.04653/) (2021)
- [Dyna-bAbI: unlocking bAbI's potential with dynamic synthetic benchmarking](https://mlsanity.com/paper/2112.00086/) (2021)
- [Improving Question Answering Performance Using Knowledge Distillation and Active Learning](https://mlsanity.com/paper/2109.12662/) (2021)
- [A Kind Introduction to Lexical and Grammatical Aspect, with a Survey of Computational Approaches](https://mlsanity.com/paper/2208.09012/) (2022)
- [POQue: Asking Participant-specific Outcome Questions for a Deeper Understanding of Complex Events](https://mlsanity.com/paper/2212.02629/) (2022)
- [Evaluating LLMs for Targeted Concept Simplification for Domain-Specific Texts](https://mlsanity.com/paper/2410.20763/) (2024)
- [Hidden Schema Networks](https://mlsanity.com/paper/2207.03777/) (2022)
- [A Survey on Measuring and Mitigating Reasoning Shortcuts in Machine Reading Comprehension](https://mlsanity.com/paper/2209.01824/) (2022)
- [Event Causality Identification with Causal News Corpus -- Shared Task 3, CASE 2022](https://mlsanity.com/paper/2211.12154/) (2022)
- [Situation and Behavior Understanding by Trope Detection on Films](https://mlsanity.com/paper/2101.07632/) (2021)
- [Reimagining Retrieval Augmented Language Models for Answering Queries](https://mlsanity.com/paper/2306.01061/) (2023)
- [Is ChatGPT the Future of Causal Text Mining? A Comprehensive Evaluation and Analysis](https://mlsanity.com/paper/2402.14484/) (2024)
- [CogME: A Cognition-Inspired Multi-Dimensional Evaluation Metric for Story Understanding](https://mlsanity.com/paper/2107.09847/) (2021)
- [IDIAPers @ Causal News Corpus 2022: Extracting Cause-Effect-Signal Triplets via Pre-trained Autoregressive Language Model](https://mlsanity.com/paper/2209.03891/) (2022)
- [On Degrees of Freedom in Defining and Testing Natural Language Understanding](https://mlsanity.com/paper/2305.15130/) (2023)
- [Design considerations for a hierarchical semantic compositional framework for medical natural language understanding](https://mlsanity.com/paper/2204.02067/) (2022)

## Abstract (arXiv)

Many tasks aim to measure machine reading comprehension (MRC), often focusing on question types presumed to be difficult. Rarely, however, do task designers start by considering what systems should in fact comprehend. In this paper we make two key contributions. First, we argue that existing approaches do not adequately define comprehension; they are too unsystematic about what content is tested. Second, we present a detailed definition of comprehension -- a "Template of Understanding" -- for a widely useful class of texts, namely short narratives. We then conduct an experiment that strongly suggests existing systems are not up to the task of narrative understanding as we define it.