# R4C: A Benchmark for Evaluating RC Systems to Get the Right Answer for the Right Reason

arXiv: https://arxiv.org/abs/1910.04601 · HTML: https://mlsanity.com/paper/1910.04601/

## Key facts

- Cited by 36 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 Oct 2019.

## Built on by (most cited first, 30 of 36)

- [ScienceWorld: Is your Agent Smarter than a 5th Grader?](https://mlsanity.com/paper/2203.07540/) (2022)
- [Explaining Answers with Entailment Trees](https://mlsanity.com/paper/2104.08661/) (2021)
- [Teach Me to Explain: A Review of Datasets for Explainable Natural Language Processing](https://mlsanity.com/paper/2102.12060/) (2021)
- [Iteratively Prompt Pre-trained Language Models for Chain of Thought](https://mlsanity.com/paper/2203.08383/) (2022)
- [Natural Language Reasoning, A Survey](https://mlsanity.com/paper/2303.14725/) (2023)
- [ExplaGraphs: An Explanation Graph Generation Task for Structured Commonsense Reasoning](https://mlsanity.com/paper/2104.07644/) (2021)
- [AR-LSAT: Investigating Analytical Reasoning of Text](https://mlsanity.com/paper/2104.06598/) (2021)
- [From LSAT: The Progress and Challenges of Complex Reasoning](https://mlsanity.com/paper/2108.00648/) (2021)
- [MoreHopQA: More Than Multi-hop Reasoning](https://mlsanity.com/paper/2406.13397/) (2024)
- [Decomposing Complex Questions Makes Multi-Hop QA Easier and More Interpretable](https://mlsanity.com/paper/2110.13472/) (2021)
- [NeedleBench: Evaluating LLM Retrieval and Reasoning Across Varying Information Densities](https://mlsanity.com/paper/2407.11963/) (2024)
- [On the Diversity and Limits of Human Explanations](https://mlsanity.com/paper/2106.11988/) (2021)
- [Position: Measure Dataset Diversity, Don't Just Claim It](https://mlsanity.com/paper/2407.08188/) (2024)
- [WikiWhy: Answering and Explaining Cause-and-Effect Questions](https://mlsanity.com/paper/2210.12152/) (2022)
- [METGEN: A Module-Based Entailment Tree Generation Framework for Answer Explanation](https://mlsanity.com/paper/2205.02593/) (2022)
- [NOAHQA: Numerical Reasoning with Interpretable Graph Question Answering Dataset](https://mlsanity.com/paper/2109.10604/) (2021)
- [Summarize-then-Answer: Generating Concise Explanations for Multi-hop Reading Comprehension](https://mlsanity.com/paper/2109.06853/) (2021)
- [On Evaluating Explanation Utility for Human-AI Decision Making in NLP](https://mlsanity.com/paper/2407.03545/) (2024)
- [Can In-context Learners Learn a Reasoning Concept from Demonstrations?](https://mlsanity.com/paper/2212.01692/) (2022)
- [Towards a Framework for Evaluating Explanations in Automated Fact Verification](https://mlsanity.com/paper/2403.20322/) (2024)
- [On the Challenges of Evaluating Compositional Explanations in Multi-Hop Inference: Relevance, Completeness, and Expert Ratings](https://mlsanity.com/paper/2109.03334/) (2021)
- [XplainLLM: A Knowledge-Augmented Dataset for Reliable Grounded Explanations in LLMs](https://mlsanity.com/paper/2311.08614/) (2023)
- [DRAGON: Domain-specific Robust Automatic Data Generation for RAG Optimization](https://mlsanity.com/paper/2505.10989/) (2025)
- [A Survey on Measuring and Mitigating Reasoning Shortcuts in Machine Reading Comprehension](https://mlsanity.com/paper/2209.01824/) (2022)
- [MetaLogic: Logical Reasoning Explanations with Fine-Grained Structure](https://mlsanity.com/paper/2210.12487/) (2022)
- [A Logical Pattern Memory Pre-trained Model for Entailment Tree Generation](https://mlsanity.com/paper/2403.06410/) (2024)
- [Analyzing the Effectiveness of the Underlying Reasoning Tasks in Multi-hop Question Answering](https://mlsanity.com/paper/2302.05963/) (2023)
- [Concept-aware Data Construction Improves In-context Learning of Language Models](https://mlsanity.com/paper/2403.09703/) (2024)
- [How Well Do Multi-hop Reading Comprehension Models Understand Date Information?](https://mlsanity.com/paper/2210.05208/) (2022)
- [A Comprehensive Survey on Multi-hop Machine Reading Comprehension Datasets and Metrics](https://mlsanity.com/paper/2212.04070/) (2022)

## Abstract (arXiv)

Recent studies have revealed that reading comprehension (RC) systems learn to exploit annotation artifacts and other biases in current datasets. This prevents the community from reliably measuring the progress of RC systems. To address this issue, we introduce R4C, a new task for evaluating RC systems' internal reasoning. R4C requires giving not only answers but also derivations: explanations that justify predicted answers. We present a reliable, crowdsourced framework for scalably annotating RC datasets with derivations. We create and publicly release the R4C dataset, the first, quality-assured dataset consisting of 4.6k questions, each of which is annotated with 3 reference derivations (i.e. 13.8k derivations). Experiments show that our automatic evaluation metrics using multiple reference derivations are reliable, and that R4C assesses different skills from an existing benchmark.