mlsanityData to 14 Oct 2026

R4C: A Benchmark for Evaluating RC Systems to Get the Right Answer for the Right Reason

arXiv 1910.04601 ↗First posted 10 Oct 2019cs.CLcited by 36

Not on the map. Why?

Abstract (arXiv)

Recent studies have revealed that reading comprehension (RC) systems learn to exploit annotation artifacts and other biases in current datasets. This prevents the community from reliably measuring the progress of RC systems. To address this issue, we introduce R4C, a new task for evaluating RC systems' internal reasoning. R4C requires giving not only answers but also derivations: explanations that justify predicted answers. We present a reliable, crowdsourced framework for scalably annotating RC datasets with derivations. We create and publicly release the R4C dataset, the first, quality-assured dataset consisting of 4.6k questions, each of which is annotated with 3 reference derivations (i.e. 13.8k derivations). Experiments show that our automatic evaluation metrics using multiple reference derivations are reliable, and that R4C assesses different skills from an existing benchmark.

Key facts. Cited by 36 papers in the map (data through 14 Oct 2026). First posted on arXiv 10 Oct 2019.

Builds on

No reference list yet.

Built on by

Cited by year

Citing papers per year: 2021: 10; 2022: 9; 2023: 6; 2024: 9; 2025: 2.

36 papers.

Sort:

By area:

  1. 2022ScienceWorld: Is your Agent Smarter than a 5th Grader?cited by 327
  2. 2021Explaining Answers with Entailment Treescited by 152
  3. 2021Teach Me to Explain: A Review of Datasets for Explainable Natural Language Processingcited by 126
  4. 2022Iteratively Prompt Pre-trained Language Models for Chain of Thoughtcited by 75
  5. 2023Natural Language Reasoning, A Surveycited by 61
  6. 2021ExplaGraphs: An Explanation Graph Generation Task for Structured Commonsense Reasoningcited by 58
  7. 2021AR-LSAT: Investigating Analytical Reasoning of Textcited by 51
  8. 2021From LSAT: The Progress and Challenges of Complex Reasoningcited by 43
  9. 2024MoreHopQA: More Than Multi-hop Reasoningcited by 37
  10. 2021Decomposing Complex Questions Makes Multi-Hop QA Easier and More Interpretablecited by 35
  11. 2024NeedleBench: Evaluating LLM Retrieval and Reasoning Across Varying Information Densitiescited by 32
  12. 2021On the Diversity and Limits of Human Explanationscited by 30
24 more
  1. 2024Position: Measure Dataset Diversity, Don't Just Claim Itcited by 26
  2. 2022WikiWhy: Answering and Explaining Cause-and-Effect Questionscited by 21
  3. 2022METGEN: A Module-Based Entailment Tree Generation Framework for Answer Explanationcited by 17
  4. 2021NOAHQA: Numerical Reasoning with Interpretable Graph Question Answering Datasetcited by 14
  5. 2021Summarize-then-Answer: Generating Concise Explanations for Multi-hop Reading Comprehensioncited by 10
  6. 2024On Evaluating Explanation Utility for Human-AI Decision Making in NLPcited by 9
  7. 2022Can In-context Learners Learn a Reasoning Concept from Demonstrations?cited by 7
  8. 2024Towards a Framework for Evaluating Explanations in Automated Fact Verificationcited by 7
  9. 2021On the Challenges of Evaluating Compositional Explanations in Multi-Hop Inference: Relevance, Completeness, and Expert Ratingscited by 5
  10. 2023XplainLLM: A Knowledge-Augmented Dataset for Reliable Grounded Explanations in LLMscited by 5
  11. 2025DRAGON: Domain-specific Robust Automatic Data Generation for RAG Optimizationcited by 5
  12. 2022A Survey on Measuring and Mitigating Reasoning Shortcuts in Machine Reading Comprehensioncited by 4
  13. 2022MetaLogic: Logical Reasoning Explanations with Fine-Grained Structurecited by 3
  14. 2024A Logical Pattern Memory Pre-trained Model for Entailment Tree Generationcited by 3
  15. 2023Analyzing the Effectiveness of the Underlying Reasoning Tasks in Multi-hop Question Answeringcited by 2
  16. 2024Concept-aware Data Construction Improves In-context Learning of Language Modelscited by 2
  17. 2022How Well Do Multi-hop Reading Comprehension Models Understand Date Information?cited by 1
  18. 2022A Comprehensive Survey on Multi-hop Machine Reading Comprehension Datasets and Metricscited by 1
  19. 2023On Degrees of Freedom in Defining and Testing Natural Language Understandingcited by 1
  20. 2023Answering Unseen Questions With Smaller Language Models Using Rationale Generation and Dense Retrievalcited by 1
  21. 2024Meta-prompting Optimized Retrieval-augmented Generationcited by 1
  22. 2025Tgea: An error-annotated dataset and benchmark tasks for text generation from pretrained language modelscited by 1
  23. 2023Contrastive Learning for Inference in Dialoguecited by 0
  24. 2024CARE: A Clue-guided Assistant for CSRs to Read User Manualscited by 0