mlsanityData to 7 Oct 2026

Adaptations of ROUGE and BLEU to Better Evaluate Machine Reading Comprehension Task

arXiv 1806.03578 ↗First posted 10 Jun 2018cs.CLcited by 20

Not on the map. Why?

Abstract (arXiv)

Current evaluation metrics to question answering based machine reading comprehension (MRC) systems generally focus on the lexical overlap between the candidate and reference answers, such as ROUGE and BLEU. However, bias may appear when these metrics are used for specific question types, especially questions inquiring yes-no opinions and entity lists. In this paper, we make adaptations on the metrics to better correlate n-gram overlap with the human judgment for answers to these two question types. Statistical analysis proves the effectiveness of our approach. Our adaptations may provide positive guidance for the development of real-scene MRC systems.

Key facts. Cited by 20 papers in the map (data through 7 Oct 2026). First posted on arXiv 10 Jun 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 3; 2023: 6; 2024: 5; 2025: 4; 2026: 1.

20 papers.

Sort:

By area:

  1. 2022Tomayto, Tomahto. Beyond Token-level Answer Equivalence for Question Answering Evaluationcited by 66
  2. 2021Semantic Answer Similarity for Evaluating Question Answering Modelscited by 47
  3. 2023Which Prompts Make The Difference? Data Prioritization For Efficient Human LLM Evaluationcited by 18
  4. 2022PVGRU: Generating Diverse and Relevant Dialogue Responses via Pseudo-Variational Mechanismcited by 6
  5. 2025Problem Solved? Information Extraction Design Space for Layout-Rich Documents using LLMscited by 6
  6. 2024It Is Not About What You Say, It Is About How You Say It: A Surprisingly Simple Approach for Improving Reading Comprehensioncited by 4
  7. 2023Learning Answer Generation using Supervision from Automatic Question Answering Evaluatorscited by 3
  8. 2023Redundancy Aware Multi-Reference Based Gainwise Evaluation of Extractive Summarizationcited by 1
  9. 2023SQUARE: Automatic Question Answering Evaluation using Multiple Positive and Negative Referencescited by 1
  10. 2024A Unified Data Augmentation Framework for Low-Resource Multi-Domain Dialogue Generationcited by 1
  11. 2024Towards Robust Evaluation: A Comprehensive Taxonomy of Datasets and Metrics for Open Domain Question Answering in the Era of Large Language Modelscited by 1
  12. 2025SMILE: A Composite Lexical-Semantic Metric for Question-Answering Evaluationcited by 1
8 more
  1. 2025SynRAG: A Large Language Model Framework for Executable Query Generation in Heterogeneous SIEM Systemcited by 1
  2. 2026UDVideoQA: A Traffic Video Question Answering Dataset for Multi-Object Spatio-Temporal Reasoning in Urban Dynamicscited by 1
  3. 2022Multi-Narrative Semantic Overlap Task: Evaluation and Benchmarkcited by 0
  4. 2023Knowing-how & Knowing-that: A New Task for Machine Comprehension of User Manualscited by 0
  5. 2023Leveraging Deep Learning for Abstractive Code Summarization of Unofficial Documentationcited by 0
  6. 2024Reliability Analysis of Psychological Concept Extraction and Classification in User-penned Textcited by 0
  7. 2024Instant Answering in E-Commerce Buyer-Seller Messaging using Message-to-Question Reformulationcited by 0
  8. 2025ACE-RLHF: Automated Code Evaluation and Socratic Feedback Generation Tool using Large Language Models and Reinforcement Learning with Human Feedbackcited by 0