Human vs Automatic Metrics: on the Importance of Correlation Design
Not on the map. Why?
Abstract (arXiv)
This paper discusses two existing approaches to the correlation analysis between automatic evaluation metrics and human scores in the area of natural language generation. Our experiments show that depending on the usage of a system- or sentence-level correlation analysis, correlation results between automatic scores and human judgments are inconsistent.
Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). First posted on arXiv 29 May 2018.
Builds on
No reference list yet.
Built on by
Citing papers per year: 2021: 1; 2022: 3; 2023: 1; 2024: 0; 2025: 0; 2026: 1.
6 papers.
- 2022Pretrained Language Models for Text Generation: A Surveycited by 121
- 2022Who Evaluates the Evaluators? On Automatic Metrics for Assessing AI-based Offensive Code Generatorscited by 10
- 2023Assessing Evaluation Metrics for Neural Test Oracle Generationcited by 7
- 2021Macro-Average: Rare Types Are Important Toocited by 4
- 2022Are metrics measuring what they should? An evaluation of image captioning task metricscited by 2
- 2026Judge Like Human Examiners: A Weighted Importance Multi-Point Evaluation Framework for Generative Tasks with Long-form Answerscited by 0
- 2026Judge Like Human Examiners: A Weighted Importance Multi-Point Evaluation Framework for Generative Tasks with Long-form Answers13 Apr 2026
- 2023Assessing Evaluation Metrics for Neural Test Oracle Generation11 Oct 2023
- 2022Who Evaluates the Evaluators? On Automatic Metrics for Assessing AI-based Offensive Code Generators12 Dec 2022
- 2022Are metrics measuring what they should? An evaluation of image captioning task metrics4 Jul 2022
- 2022Pretrained Language Models for Text Generation: A Survey14 Jan 2022
- 2021Macro-Average: Rare Types Are Important Too12 Apr 2021