Data to 7 Oct 2026

The price of debiasing automatic metrics in natural language evaluation

arXiv 1807.02202 ↗First posted 6 Jul 2018cs.CLcited by 60

Not on the map. Why?

Abstract (arXiv)

For evaluating generation systems, automatic metrics such as BLEU cost nothing to run but have been shown to correlate poorly with human judgment, leading to systematic bias against certain model improvements. On the other hand, averaging human judgments, the unbiased gold standard, is often too expensive. In this paper, we use control variates to combine automatic metrics with human evaluation to obtain an unbiased estimator with lower cost than human evaluation alone. In practice, however, we obtain only a 7-13% cost reduction on evaluating summarization and open-response question answering systems. We then prove that our estimator is optimal: there is no unbiased estimator with lower cost. Our theory further highlights the two fundamental bottlenecks---the automatic metric and the prompt shown to human evaluators---both of which need to be improved to obtain greater cost savings.

Key facts. Cited by 60 papers in the map (data through 7 Oct 2026). First posted on arXiv 6 Jul 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2020: 1; 2021: 15; 2022: 10; 2023: 8; 2024: 12; 2025: 7; 2026: 7.

60 papers.

Sort:

By area:

2020Learning to summarize from human feedbackcited by 2,562
2023Can Large Language Models Transform Computational Social Science?cited by 245
2022Revisiting the Gold Standard: Grounding Summarization Evaluation with Robust Human Evaluationcited by 141
2023CoAnnotating: Uncertainty-Guided Work Allocation between Human and Large Language Models for Data Annotationcited by 93
2022An Empirical Survey on Long Document Summarization: Datasets, Models and Metricscited by 74
2021IndoNLG: Benchmark and Resources for Evaluating Indonesian Natural Language Generationcited by 57
2023Bridging the Gap: A Survey on Integrating (Human) Feedback for Natural Language Generationcited by 55
2021GooAQ: Open Question Answering with Diverse Answer Typescited by 53
2024AutoEval Done Right: Using Synthetic Data for Model Evaluationcited by 53
2024Limits to scalable evaluation at the frontier: LLM as Judge won't beat twice the datacited by 46
2021Can Audio Captions Be Evaluated with Image Caption Metrics?cited by 33
2022How Far are We from Robust Long Abstractive Summarization?cited by 25

38 more

2025Safer or Luckier? LLMs as Safety Evaluators Are Not Robust to Artifactscited by 25
2024Stratified Prediction-Powered Inference for Hybrid Language Model Evaluationcited by 24
2021Unsupervised Extractive Summarization using Pointwise Mutual Informationcited by 23
2021A Training-free and Reference-free Summarization Evaluation Metric via Centrality-weighted Relevance and Self-referenced Redundancycited by 21
2021GENIE: Toward Reproducible and Standardized Human Evaluation for Text Generationcited by 20
2022Why is constrained neural language generation particularly challenging?cited by 19
2021The statistical advantage of automatic NLG metrics at the system levelcited by 18
2023Which Prompts Make The Difference? Data Prioritization For Efficient Human LLM Evaluationcited by 18
2025Cost-Optimal Active AI Model Evaluationcited by 14
2021Answer Generation for Retrieval-based Question Answering Systemscited by 13
2022Quiz Design Task: Helping Teachers Create Quizzes with Automated Question Generationcited by 12
2022Toward More Effective Human Evaluation for Machine Translationcited by 10
2023Reinforcement Learning for Generative AI: State of the Art, Opportunities and Open Research Challengescited by 10
2024Decoding the Narratives: Analyzing Personal Drug Experiences Shared on Redditcited by 9
2025Adaptive Prediction-Powered AutoEval with Reliability and Efficiency Guaranteescited by 9
2026Multiple-Prediction-Powered Inferencecited by 9
2023Evaluation of Question Generation Needs More Referencescited by 8
2025No Free Lunch: Non-Asymptotic Analysis of Prediction-Powered Inferencecited by 8
2021How to Evaluate a Summarizer: Study Design and Statistical Analysis for Manual Linguistic Quality Evaluationcited by 7
2021Language Model Augmented Relevance Scorecited by 7
2021Cross-Lingual Open-Domain Question Answering with Answer Sentence Generationcited by 6
2025Accelerating Unbiased LLM Evaluation via Synthetic Feedbackcited by 6
2021Naturalness Evaluation of Natural Language Generation in Task-oriented Dialogues using BERTcited by 5
2021Using Sampling to Estimate and Improve Performance of Automated Scoring Systems with Guaranteescited by 5
2024Bridging Research and Readers: A Multi-Modal Automated Academic Papers Interpretation Systemcited by 5
2022On the Effectiveness of Automated Metrics for Text Generation Systemscited by 4
2024SumRecom: A Personalized Summarization Approach by Learning from Users' Feedbackcited by 4
2022Searching for a higher power in the human evaluation of MTcited by 3
2023Correction of Errors in Preference Ratings from Automated Metrics for Text Generationcited by 3
2021Dynamic Human Evaluation for Relative Model Comparisonscited by 2
2022COMPASS: a Creative Support System that Alerts Novelists to the Unnoticed Missing Contentscited by 2
2024Benchmarking LLMs on the Semantic Overlap Summarization Taskcited by 2
2024Favi-Score: A Measure for Favoritism in Automated Preference Ratings for Generative AI Evaluationcited by 2
2024A Critical Look at Meta-evaluating Summarisation Evaluation Metricscited by 2
2024An Extensive Evaluation of Factual Consistency in Large Language Models for Data-to-Text Generationcited by 2
2024A Measure of the System Dependence of Automated Metricscited by 2
2026Who Watches the Watchmen? Humans Disagree With Translation Metrics on Unseen Domainscited by 2
2021Context Matters in Semantically Controlled Language Generation for Task-oriented Dialogue Systemscited by 1

50 most cited of 60 All 60 (JSON)