# A Framework for Evaluation of Machine Reading Comprehension Gold Standards

arXiv: https://arxiv.org/abs/2003.04642 · HTML: https://mlsanity.com/paper/2003.04642/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 Mar 2020.

## Built on by (most cited first, 18 of 18)

- [FLASK: Fine-grained Language Model Evaluation based on Alignment Skill Sets](https://mlsanity.com/paper/2307.10928/) (2023)
- [QA Dataset Explosion: A Taxonomy of NLP Resources for Question Answering and Reading Comprehension](https://mlsanity.com/paper/2107.12708/) (2021)
- [Comparing Test Sets with Item Response Theory](https://mlsanity.com/paper/2106.00840/) (2021)
- [It Takes Two to Tango: Navigating Conceptualizations of NLP Tasks and Measurements of Performance](https://mlsanity.com/paper/2305.09022/) (2023)
- [Machine Reading, Fast and Slow: When Do Models "Understand" Language?](https://mlsanity.com/paper/2209.07430/) (2022)
- [Do Natural Language Explanations Represent Valid Logical Arguments? Verifying Entailment in Explainable NLI Gold Standards](https://mlsanity.com/paper/2105.01974/) (2021)
- [Seemingly Plausible Distractors in Multi-Hop Reasoning: Are Large Language Models Attentive Readers?](https://mlsanity.com/paper/2409.05197/) (2024)
- [Can Transformers Reason in Fragments of Natural Language?](https://mlsanity.com/paper/2211.05417/) (2022)
- [WLASL-LEX: a Dataset for Recognising Phonological Properties in American Sign Language](https://mlsanity.com/paper/2203.06096/) (2022)
- [A Survey on Measuring and Mitigating Reasoning Shortcuts in Machine Reading Comprehension](https://mlsanity.com/paper/2209.01824/) (2022)
- [Feeding What You Need by Understanding What You Learned](https://mlsanity.com/paper/2203.02753/) (2022)
- [Pay Attention to Real World Perturbations! Natural Robustness Evaluation in Machine Reading Comprehension](https://mlsanity.com/paper/2502.16523/) (2025)
- [On Degrees of Freedom in Defining and Testing Natural Language Understanding](https://mlsanity.com/paper/2305.15130/) (2023)
- [A Systematic Survey of Natural Language Processing for the Greek Language](https://mlsanity.com/paper/2407.09861/) (2024)
- [Natural Context Drift Undermines the Natural Language Understanding of Large Language Models](https://mlsanity.com/paper/2509.01093/) (2025)
- [An Expert Schema for Evaluating Large Language Model Errors in Scholarly Question-Answering Systems](https://mlsanity.com/paper/2602.21059/) (2026)
- [Investigating a Benchmark for Training-set free Evaluation of Linguistic Capabilities in Machine Reading Comprehension](https://mlsanity.com/paper/2408.05023/) (2024)
- [MRCEval: A Comprehensive, Challenging and Accessible Machine Reading Comprehension Benchmark](https://mlsanity.com/paper/2503.07144/) (2025)

## Abstract (arXiv)

Machine Reading Comprehension (MRC) is the task of answering a question over a paragraph of text. While neural MRC systems gain popularity and achieve noticeable performance, issues are being raised with the methodology used to establish their performance, particularly concerning the data design of gold standards that are used to evaluate them. There is but a limited understanding of the challenges present in this data, which makes it hard to draw comparisons and formulate reliable hypotheses. As a first step towards alleviating the problem, this paper proposes a unifying framework to systematically investigate the present linguistic features, required reasoning and background knowledge and factual correctness on one hand, and the presence of lexical cues as a lower bound for the requirement of understanding on the other hand. We propose a qualitative annotation schema for the first and a set of approximative metrics for the latter. In a first application of the framework, we analyse modern MRC gold standards and present our findings: the absence of features that contribute towards lexical ambiguity, the varying factual correctness of the expected answers and the presence of lexical cues, all of which potentially lower the reading comprehension complexity and quality of the evaluation data.