# Analyzing Multiple-Choice Reading and Listening Comprehension Tests

arXiv: https://arxiv.org/abs/2307.01076 · HTML: https://mlsanity.com/paper/2307.01076/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 30 references that are in the map.
- First posted on arXiv 3 Jul 2023.

## Builds on (most cited first, 22 of 22)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge](https://mlsanity.com/paper/1803.05457/) (2018)
- [HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering](https://mlsanity.com/paper/1809.09600/) (2018)
- [Big Bird: Transformers for Longer Sequences](https://mlsanity.com/paper/2007.14062/) (2020)
- [Know What You Don't Know: Unanswerable Questions for SQuAD](https://mlsanity.com/paper/1806.03822/) (2018)
- [RACE: Large-scale ReAding Comprehension Dataset From Examinations](https://mlsanity.com/paper/1704.04683/) (2017)
- [Adversarial Examples for Evaluating Reading Comprehension Systems](https://mlsanity.com/paper/1707.07328/) (2017)
- [NewsQA: A Machine Comprehension Dataset](https://mlsanity.com/paper/1611.09830/) (2016)
- [LUKE: Deep Contextualized Entity Representations with Entity-aware Self-attention](https://mlsanity.com/paper/2010.01057/) (2020)
- [Cosmos QA: Machine Reading Comprehension with Contextual Commonsense Reasoning](https://mlsanity.com/paper/1909.00277/) (2019)
- [ReClor: A Reading Comprehension Dataset Requiring Logical Reasoning](https://mlsanity.com/paper/2002.04326/) (2020)
- [DREAM: A Challenge Dataset and Models for Dialogue-Based Reading Comprehension](https://mlsanity.com/paper/1902.00164/) (2019)
- [How Much Reading Does Reading Comprehension Require? A Critical Investigation of Popular Benchmarks](https://mlsanity.com/paper/1808.04926/) (2018)
- [Retrospective Reader for Machine Reading Comprehension](https://mlsanity.com/paper/2001.09694/) (2020)
- [Assessing the Benchmarking Capacity of Machine Reading Comprehension Datasets](https://mlsanity.com/paper/1911.09241/) (2019)
- [Logic-Driven Context Extension and Data Augmentation for Logical Reasoning of Text](https://mlsanity.com/paper/2105.03659/) (2021)
- [What does BERT Learn from Multiple-Choice Reading Comprehension Datasets?](https://mlsanity.com/paper/1910.12391/) (2019)
- [Supervised and Unsupervised Transfer Learning for Question Answering](https://mlsanity.com/paper/1711.05345/) (2017)
- [Towards Machine Comprehension of Spoken Content: Initial TOEFL Listening Comprehension Test by Machine](https://mlsanity.com/paper/1608.06378/) (2016)
- [Multiple-Choice Question Generation: Towards an Automated Assessment Framework](https://mlsanity.com/paper/2209.11830/) (2022)
- [World Knowledge in Multiple Choice Reading Comprehension](https://mlsanity.com/paper/2211.07040/) (2022)
- [An Initial Investigation of Non-Native Spoken Question-Answering](https://mlsanity.com/paper/2107.04691/) (2021)

## Built on by (most cited first, 4 of 4)

- [Do LLMs Give Psychometrically Plausible Responses in Educational Assessments?](https://mlsanity.com/paper/2506.09796/) (2025)
- [Automatic Generation and Evaluation of Reading Comprehension Test Items with Large Language Models](https://mlsanity.com/paper/2404.07720/) (2024)
- [An Information-Theoretic Approach to Analyze NLP Classification Tasks](https://mlsanity.com/paper/2402.00978/) (2024)
- [SP-MCQA: Evaluating Intelligibility of TTS Beyond the Word Level](https://mlsanity.com/paper/2510.26190/) (2025)

## Abstract (arXiv)

Multiple-choice reading and listening comprehension tests are an important part of language assessment. Content creators for standard educational tests need to carefully curate questions that assess the comprehension abilities of candidates taking the tests. However, recent work has shown that a large number of questions in general multiple-choice reading comprehension datasets can be answered without comprehension, by leveraging world knowledge instead. This work investigates how much of a contextual passage needs to be read in multiple-choice reading based on conversation transcriptions and listening comprehension tests to be able to work out the correct answer. We find that automated reading comprehension systems can perform significantly better than random with partial or even no access to the context passage. These findings offer an approach for content creators to automatically capture the trade-off between comprehension and world knowledge required for their proposed questions.