# What does BERT Learn from Multiple-Choice Reading Comprehension Datasets?

arXiv: https://arxiv.org/abs/1910.12391 · HTML: https://mlsanity.com/paper/1910.12391/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 Oct 2019.

## Built on by (most cited first, 30 of 30)

- [Prompting GPT-3 To Be Reliable](https://mlsanity.com/paper/2210.09150/) (2022)
- [What Will it Take to Fix Benchmarking in Natural Language Understanding?](https://mlsanity.com/paper/2104.02145/) (2021)
- [Shortcut Learning of Large Language Models in Natural Language Understanding](https://mlsanity.com/paper/2208.11857/) (2022)
- [Towards Interpreting and Mitigating Shortcut Learning Behavior of NLU Models](https://mlsanity.com/paper/2103.06922/) (2021)
- [Why Machine Reading Comprehension Models Learn Shortcuts?](https://mlsanity.com/paper/2106.01024/) (2021)
- [Using Adversarial Attacks to Reveal the Statistical Bias in Machine Reading Comprehension Models](https://mlsanity.com/paper/2105.11136/) (2021)
- [It Takes Two to Tango: Navigating Conceptualizations of NLP Tasks and Measurements of Performance](https://mlsanity.com/paper/2305.09022/) (2023)
- [Inspecting the concept knowledge graph encoded by modern language models](https://mlsanity.com/paper/2105.13471/) (2021)
- [BiRdQA: A Bilingual Dataset for Question Answering on Tricky Riddles](https://mlsanity.com/paper/2109.11087/) (2021)
- [Shaking Syntactic Trees on the Sesame Street: Multilingual Probing with Controllable Perturbations](https://mlsanity.com/paper/2109.14017/) (2021)
- [Navigating the Shortcut Maze: A Comprehensive Analysis of Shortcut Learning in Text Classification by Language Models](https://mlsanity.com/paper/2409.17455/) (2024)
- [Leaving the barn door open for Clever Hans: Simple features predict LLM benchmark answers](https://mlsanity.com/paper/2410.11672/) (2024)
- [Desiderata for the Context Use of Question Answering Systems](https://mlsanity.com/paper/2401.18001/) (2024)
- [World Knowledge in Multiple Choice Reading Comprehension](https://mlsanity.com/paper/2211.07040/) (2022)
- [Down and Across: Introducing Crossword-Solving as a New NLP Benchmark](https://mlsanity.com/paper/2205.10442/) (2022)
- [A Survey on Measuring and Mitigating Reasoning Shortcuts in Machine Reading Comprehension](https://mlsanity.com/paper/2209.01824/) (2022)
- [Analyzing Multiple-Choice Reading and Listening Comprehension Tests](https://mlsanity.com/paper/2307.01076/) (2023)
- [It Is Not About What You Say, It Is About How You Say It: A Surprisingly Simple Approach for Improving Reading Comprehension](https://mlsanity.com/paper/2406.16779/) (2024)
- [Sometimes We Want Translationese](https://mlsanity.com/paper/2104.07623/) (2021)
- [PALRACE: Reading Comprehension Dataset with Human Data and Labeled Rationales](https://mlsanity.com/paper/2106.12373/) (2021)
- [Investigating Ensemble Methods for Model Robustness Improvement of Text Classifiers](https://mlsanity.com/paper/2210.16298/) (2022)
- [Unreasonable Effectiveness of Rule-Based Heuristics in Solving Russian SuperGLUE Tasks](https://mlsanity.com/paper/2105.01192/) (2021)
- [Coarse-to-Careful: Seeking Semantic-related Knowledge for Open-domain Commonsense Question Answering](https://mlsanity.com/paper/2107.01592/) (2021)
- [A Framework for Rationale Extraction for Deep QA models](https://mlsanity.com/paper/2110.04620/) (2021)
- [How Does Data Corruption Affect Natural Language Understanding Models? A Study on GLUE datasets](https://mlsanity.com/paper/2201.04467/) (2022)
- [EMBRACE: Evaluation and Modifications for Boosting RACE](https://mlsanity.com/paper/2305.08433/) (2023)
- [Student Answer Forecasting: Transformer-Driven Answer Choice Prediction for Language Learning](https://mlsanity.com/paper/2405.20079/) (2024)
- [Implicit Word Reordering with Knowledge Distillation for Cross-Lingual Dependency Parsing](https://mlsanity.com/paper/2502.17308/) (2025)
- [Can LLMs Solve My Grandma's Riddle? Evaluating Multilingual Large Language Models on Reasoning Traditional Bangla Tricky Riddles](https://mlsanity.com/paper/2512.20324/) (2025)
- [Your Benchmark Is Not Saturated: Reviving Multiple-Choice Evaluation with Answer Pooling](https://mlsanity.com/paper/2609.37494/) (2026)

## Abstract (arXiv)

Multiple-Choice Reading Comprehension (MCRC) requires the model to read the passage and question, and select the correct answer among the given options. Recent state-of-the-art models have achieved impressive performance on multiple MCRC datasets. However, such performance may not reflect the model's true ability of language understanding and reasoning. In this work, we adopt two approaches to investigate what BERT learns from MCRC datasets: 1) an un-readable data attack, in which we add keywords to confuse BERT, leading to a significant performance drop; and 2) an un-answerable data training, in which we train BERT on partial or shuffled input. Under un-answerable data training, BERT achieves unexpectedly high performance. Based on our experiments on the 5 key MCRC datasets - RACE, MCTest, MCScript, MCScript2.0, DREAM - we observe that 1) fine-tuned BERT mainly learns how keywords lead to correct prediction, instead of learning semantic understanding and reasoning; and 2) BERT does not need correct syntactic information to solve the task; 3) there exists artifacts in these datasets such that they can be solved even without the full context.