# EMBRACE: Evaluation and Modifications for Boosting RACE

arXiv: https://arxiv.org/abs/2305.08433 · HTML: https://mlsanity.com/paper/2305.08433/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 27 of its 50 references that are in the map.
- First posted on arXiv 15 May 2023.

## Builds on (most cited first, 27 of 27)

- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [RACE: Large-scale ReAding Comprehension Dataset From Examinations](https://mlsanity.com/paper/1704.04683/) (2017)
- [CoQA: A Conversational Question Answering Challenge](https://mlsanity.com/paper/1808.07042/) (2018)
- [QuAC : Question Answering in Context](https://mlsanity.com/paper/1808.07036/) (2018)
- [NewsQA: A Machine Comprehension Dataset](https://mlsanity.com/paper/1611.09830/) (2016)
- [Fantastic Questions and Where to Find Them: FairytaleQA -- An Authentic Dataset for Narrative Comprehension](https://mlsanity.com/paper/2203.13947/) (2022)
- [A Thorough Examination of the CNN/Daily Mail Reading Comprehension Task](https://mlsanity.com/paper/1606.02858/) (2016)
- [UnifiedQA-v2: Stronger Generalization via Broader Cross-Format Training](https://mlsanity.com/paper/2202.12359/) (2022)
- [What does BERT Learn from Multiple-Choice Reading Comprehension Datasets?](https://mlsanity.com/paper/1910.12391/) (2019)
- [Using Adversarial Attacks to Reveal the Statistical Bias in Machine Reading Comprehension Models](https://mlsanity.com/paper/2105.11136/) (2021)
- [Dual Co-Matching Network for Multi-choice Reading Comprehension](https://mlsanity.com/paper/1901.09381/) (2019)
- [Automatic Distractor Generation for Multiple Choice Questions in Standard Tests](https://mlsanity.com/paper/2011.13100/) (2020)
- [Generating Distractors for Reading Comprehension Questions from Real Examinations](https://mlsanity.com/paper/1809.02768/) (2018)
- [Quiz-Style Question Generation for News Stories](https://mlsanity.com/paper/2102.09094/) (2021)
- [Multiple-Choice Question Generation: Towards an Automated Assessment Framework](https://mlsanity.com/paper/2209.11830/) (2022)
- [A BERT-based Distractor Generation Scheme with Multi-tasking and Negative Answer Training Strategies](https://mlsanity.com/paper/2010.05384/) (2020)
- [Improving Machine Reading Comprehension with General Reading Strategies](https://mlsanity.com/paper/1810.13441/) (2018)
- [STARC: Structured Annotations for Reading Comprehension](https://mlsanity.com/paper/2004.14797/) (2020)
- [Better Distractions: Transformer-based Distractor Generation and Multiple Choice Question Filtering](https://mlsanity.com/paper/2010.09598/) (2020)
- [EQG-RACE: Examination-Type Question Generation](https://mlsanity.com/paper/2012.06106/) (2020)
- [Evidence Sentence Extraction for Machine Reading Comprehension](https://mlsanity.com/paper/1902.08852/) (2019)
- [Option Comparison Network for Multiple-choice Reading Comprehension](https://mlsanity.com/paper/1903.03033/) (2019)
- [Co-Attention Hierarchical Network: Generating Coherent Long Distractors for Reading Comprehension](https://mlsanity.com/paper/1911.08648/) (2019)
- [A Co-Matching Model for Multi-choice Reading Comprehension](https://mlsanity.com/paper/1806.04068/) (2018)
- [BERT-based distractor generation for Swedish reading comprehension questions using a small-scale dataset](https://mlsanity.com/paper/2108.03973/) (2021)
- [World Knowledge in Multiple Choice Reading Comprehension](https://mlsanity.com/paper/2211.07040/) (2022)
- [PALRACE: Reading Comprehension Dataset with Human Data and Labeled Rationales](https://mlsanity.com/paper/2106.12373/) (2021)

## Abstract (arXiv)

When training and evaluating machine reading comprehension models, it is very important to work with high-quality datasets that are also representative of real-world reading comprehension tasks. This requirement includes, for instance, having questions that are based on texts of different genres and require generating inferences or reflecting on the reading material. In this article we turn our attention to RACE, a dataset of English texts and corresponding multiple-choice questions (MCQs). Each MCQ consists of a question and four alternatives (of which one is the correct answer). RACE was constructed by Chinese teachers of English for human reading comprehension and is widely used as training material for machine reading comprehension models. By construction, RACE should satisfy the aforementioned quality requirements and the purpose of this article is to check whether they are indeed satisfied. We provide a detailed analysis of the test set of RACE for high-school students (1045 texts and 3498 corresponding MCQs) including (1) an evaluation of the difficulty of each MCQ and (2) annotations for the relevant pieces of the texts (called "bases") that are used to justify the plausibility of each alternative. A considerable number of MCQs appear not to fulfill basic requirements for this type of reading comprehension tasks, so we additionally identify the high-quality subset of the evaluated RACE corpus. We also demonstrate that the distribution of the positions of the bases for the alternatives is biased towards certain parts of texts, which is not necessarily desirable when evaluating MCQ answering and generation models.