# A Thorough Examination of the CNN/Daily Mail Reading Comprehension Task

arXiv: https://arxiv.org/abs/1606.02858 · HTML: https://mlsanity.com/paper/1606.02858/

## Key facts

- Cited by 72 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 Jun 2016.

## Built on by (most cited first, 30 of 72)

- [Large Language Models: A Survey](https://mlsanity.com/paper/2402.06196/) (2024)
- [WANLI: Worker and AI Collaboration for Natural Language Inference Dataset Creation](https://mlsanity.com/paper/2201.05955/) (2022)
- [Are We Done with MMLU?](https://mlsanity.com/paper/2406.04127/) (2024)
- [Impact of Pretraining Term Frequencies on Few-Shot Reasoning](https://mlsanity.com/paper/2202.07206/) (2022)
- [CommonsenseQA 2.0: Exposing the Limits of AI through Gamification](https://mlsanity.com/paper/2201.05320/) (2022)
- [Competency Problems: On Finding and Removing Artifacts in Language Data](https://mlsanity.com/paper/2104.08646/) (2021)
- [Asymmetry in Low-Rank Adapters of Foundation Models](https://mlsanity.com/paper/2402.16842/) (2024)
- [An Empirical Survey on Long Document Summarization: Datasets, Models and Metrics](https://mlsanity.com/paper/2207.00939/) (2022)
- [Compressing LLMs: The Truth is Rarely Pure and Never Simple](https://mlsanity.com/paper/2310.01382/) (2023)
- [Generating Data to Mitigate Spurious Correlations in Natural Language Inference Datasets](https://mlsanity.com/paper/2203.12942/) (2022)
- [Why Machine Reading Comprehension Models Learn Shortcuts?](https://mlsanity.com/paper/2106.01024/) (2021)
- [Models in the Loop: Aiding Crowdworkers with Generative Annotation Assistants](https://mlsanity.com/paper/2112.09062/) (2021)
- [Explanation-based Finetuning Makes Models More Robust to Spurious Cues](https://mlsanity.com/paper/2305.04990/) (2023)
- [From Low Rank Gradient Subspace Stabilization to Low-Rank Weights: Observations, Theories, and Applications](https://mlsanity.com/paper/2407.11239/) (2024)
- [FFN-SkipLLM: A Hidden Gem for Autoregressive Decoding with Adaptive Feed Forward Skipping](https://mlsanity.com/paper/2404.03865/) (2024)
- [Saving Dense Retriever from Shortcut Dependency in Conversational Search](https://mlsanity.com/paper/2202.07280/) (2022)
- [On the Robustness of Reading Comprehension Models to Entity Renaming](https://mlsanity.com/paper/2110.08555/) (2021)
- [TVShowGuess: Character Comprehension in Stories as Speaker Guessing](https://mlsanity.com/paper/2204.07721/) (2022)
- [Neural ranking models for document retrieval](https://mlsanity.com/paper/2102.11903/) (2021)
- [How well do you know your summarization datasets?](https://mlsanity.com/paper/2106.11388/) (2021)
- [Machine Reading, Fast and Slow: When Do Models "Understand" Language?](https://mlsanity.com/paper/2209.07430/) (2022)
- [GLoRE: Evaluating Logical Reasoning of Large Language Models](https://mlsanity.com/paper/2310.09107/) (2023)
- [Mitigating the Position Bias of Transformer Models in Passage Re-Ranking](https://mlsanity.com/paper/2101.06980/) (2021)
- [A Survey of Machine Narrative Reading Comprehension Assessments](https://mlsanity.com/paper/2205.00299/) (2022)
- [Paraphrase and Aggregate with Large Language Models for Minimizing Intent Classification Errors](https://mlsanity.com/paper/2406.17163/) (2024)
- [Does Synthetic Data Make Large Language Models More Efficient?](https://mlsanity.com/paper/2310.07830/) (2023)
- [WLASL-LEX: a Dataset for Recognising Phonological Properties in American Sign Language](https://mlsanity.com/paper/2203.06096/) (2022)
- [CNewSum: A Large-scale Chinese News Summarization Dataset with Human-annotated Adequacy and Deducibility Level](https://mlsanity.com/paper/2110.10874/) (2021)
- [Data Augmentation for Biomedical Factoid Question Answering](https://mlsanity.com/paper/2204.04711/) (2022)
- [On Evaluation of Document Classification using RVL-CDIP](https://mlsanity.com/paper/2306.12550/) (2023)

## Abstract (arXiv)

Enabling a computer to understand a document so that it can answer comprehension questions is a central, yet unsolved goal of NLP. A key factor impeding its solution by machine learned systems is the limited availability of human-annotated data. Hermann et al. (2015) seek to solve this problem by creating over a million training examples by pairing CNN and Daily Mail news articles with their summarized bullet points, and show that a neural network can then be trained to give good performance on this task. In this paper, we conduct a thorough examination of this new reading comprehension task. Our primary aim is to understand what depth of language understanding is required to do well on this task. We approach this from one side by doing a careful hand-analysis of a small subset of the problems and from the other by showing that simple, carefully designed systems can obtain accuracies of 73.6% and 76.6% on these two datasets, exceeding current state-of-the-art results by 7-10% and approaching what we believe is the ceiling for performance on this task.