# Adaptations of ROUGE and BLEU to Better Evaluate Machine Reading Comprehension Task

arXiv: https://arxiv.org/abs/1806.03578 · HTML: https://mlsanity.com/paper/1806.03578/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 Jun 2018.

## Built on by (most cited first, 20 of 20)

- [Tomayto, Tomahto. Beyond Token-level Answer Equivalence for Question Answering Evaluation](https://mlsanity.com/paper/2202.07654/) (2022)
- [Semantic Answer Similarity for Evaluating Question Answering Models](https://mlsanity.com/paper/2108.06130/) (2021)
- [Which Prompts Make The Difference? Data Prioritization For Efficient Human LLM Evaluation](https://mlsanity.com/paper/2310.14424/) (2023)
- [PVGRU: Generating Diverse and Relevant Dialogue Responses via Pseudo-Variational Mechanism](https://mlsanity.com/paper/2212.09086/) (2022)
- [Problem Solved? Information Extraction Design Space for Layout-Rich Documents using LLMs](https://mlsanity.com/paper/2502.18179/) (2025)
- [It Is Not About What You Say, It Is About How You Say It: A Surprisingly Simple Approach for Improving Reading Comprehension](https://mlsanity.com/paper/2406.16779/) (2024)
- [Learning Answer Generation using Supervision from Automatic Question Answering Evaluators](https://mlsanity.com/paper/2305.15344/) (2023)
- [Redundancy Aware Multi-Reference Based Gainwise Evaluation of Extractive Summarization](https://mlsanity.com/paper/2308.02270/) (2023)
- [SQUARE: Automatic Question Answering Evaluation using Multiple Positive and Negative References](https://mlsanity.com/paper/2309.12250/) (2023)
- [A Unified Data Augmentation Framework for Low-Resource Multi-Domain Dialogue Generation](https://mlsanity.com/paper/2406.09881/) (2024)
- [Towards Robust Evaluation: A Comprehensive Taxonomy of Datasets and Metrics for Open Domain Question Answering in the Era of Large Language Models](https://mlsanity.com/paper/2406.13232/) (2024)
- [SMILE: A Composite Lexical-Semantic Metric for Question-Answering Evaluation](https://mlsanity.com/paper/2511.17432/) (2025)
- [SynRAG: A Large Language Model Framework for Executable Query Generation in Heterogeneous SIEM System](https://mlsanity.com/paper/2512.24571/) (2025)
- [UDVideoQA: A Traffic Video Question Answering Dataset for Multi-Object Spatio-Temporal Reasoning in Urban Dynamics](https://mlsanity.com/paper/2602.21137/) (2026)
- [Multi-Narrative Semantic Overlap Task: Evaluation and Benchmark](https://mlsanity.com/paper/2201.05294/) (2022)
- [Knowing-how & Knowing-that: A New Task for Machine Comprehension of User Manuals](https://mlsanity.com/paper/2306.04187/) (2023)
- [Leveraging Deep Learning for Abstractive Code Summarization of Unofficial Documentation](https://mlsanity.com/paper/2310.15015/) (2023)
- [Reliability Analysis of Psychological Concept Extraction and Classification in User-penned Text](https://mlsanity.com/paper/2401.06709/) (2024)
- [Instant Answering in E-Commerce Buyer-Seller Messaging using Message-to-Question Reformulation](https://mlsanity.com/paper/2401.09785/) (2024)
- [ACE-RLHF: Automated Code Evaluation and Socratic Feedback Generation Tool using Large Language Models and Reinforcement Learning with Human Feedback](https://mlsanity.com/paper/2504.04657/) (2025)

## Abstract (arXiv)

Current evaluation metrics to question answering based machine reading comprehension (MRC) systems generally focus on the lexical overlap between the candidate and reference answers, such as ROUGE and BLEU. However, bias may appear when these metrics are used for specific question types, especially questions inquiring yes-no opinions and entity lists. In this paper, we make adaptations on the metrics to better correlate n-gram overlap with the human judgment for answers to these two question types. Statistical analysis proves the effectiveness of our approach. Our adaptations may provide positive guidance for the development of real-scene MRC systems.