# PALRACE: Reading Comprehension Dataset with Human Data and Labeled Rationales

arXiv: https://arxiv.org/abs/2106.12373 · HTML: https://mlsanity.com/paper/2106.12373/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 14 references that are in the map.
- First posted on arXiv 23 Jun 2021.

## Builds on (most cited first, 11 of 11)

- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [ALBERT: A Lite BERT for Self-supervised Learning of Language Representations](https://mlsanity.com/paper/1909.11942/) (2019)
- [HuggingFace's Transformers: State-of-the-art Natural Language Processing](https://mlsanity.com/paper/1910.03771/) (2019)
- [RACE: Large-scale ReAding Comprehension Dataset From Examinations](https://mlsanity.com/paper/1704.04683/) (2017)
- [Annotation Artifacts in Natural Language Inference Data](https://mlsanity.com/paper/1803.02324/) (2018)
- [ERASER: A Benchmark to Evaluate Rationalized NLP Models](https://mlsanity.com/paper/1911.03429/) (2019)
- [Rationalizing Neural Predictions](https://mlsanity.com/paper/1606.04155/) (2016)
- [Rationale-Augmented Convolutional Neural Networks for Text Classification](https://mlsanity.com/paper/1605.04469/) (2016)
- [What does BERT Learn from Multiple-Choice Reading Comprehension Datasets?](https://mlsanity.com/paper/1910.12391/) (2019)
- [Benchmarking Robustness of Machine Reading Comprehension Models](https://mlsanity.com/paper/2004.14004/) (2020)
- [STARC: Structured Annotations for Reading Comprehension](https://mlsanity.com/paper/2004.14797/) (2020)

## Built on by (most cited first, 3 of 3)

- [Human Attention during Goal-directed Reading Comprehension Relies on Task Optimization](https://mlsanity.com/paper/2107.05799/) (2021)
- [EMBRACE: Evaluation and Modifications for Boosting RACE](https://mlsanity.com/paper/2305.08433/) (2023)
- [Response-free item difficulty modelling for multiple-choice items with fine-tuned transformers: Component-wise representation and multi-task learning](https://mlsanity.com/paper/2605.16991/) (2026)

## Abstract (arXiv)

Pre-trained language models achieves high performance on machine reading comprehension (MRC) tasks but the results are hard to explain. An appealing approach to make models explainable is to provide rationales for its decision. To investigate whether human rationales can further improve current models and to facilitate supervised learning of human rationales, here we present PALRACE (Pruned And Labeled RACE), a new MRC dataset with human labeled rationales for 800 passages selected from the RACE dataset. We further classified the question to each passage into 6 types. Each passage was read by at least 26 human readers, who labeled their rationales to answer the question. It is demonstrated that models such as RoBERTa-large outperforms human readers in all 6 types of questions, including inference questions, but its performance can be further improved when having access to the human rationales. Simpler models and pre-trained models that are not fine-tuned based on the task benefit more from human rationales, and their performance can be boosted by more than 30% by rationales. With access to human rationales, a simple model based on the GloVe word embedding can reach the performance of BERT-base.