# Human vs Automatic Metrics: on the Importance of Correlation Design

arXiv: https://arxiv.org/abs/1805.11474 · HTML: https://mlsanity.com/paper/1805.11474/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 May 2018.

## Built on by (most cited first, 6 of 6)

- [Pretrained Language Models for Text Generation: A Survey](https://mlsanity.com/paper/2201.05273/) (2022)
- [Who Evaluates the Evaluators? On Automatic Metrics for Assessing AI-based Offensive Code Generators](https://mlsanity.com/paper/2212.06008/) (2022)
- [Assessing Evaluation Metrics for Neural Test Oracle Generation](https://mlsanity.com/paper/2310.07856/) (2023)
- [Macro-Average: Rare Types Are Important Too](https://mlsanity.com/paper/2104.05700/) (2021)
- [Are metrics measuring what they should? An evaluation of image captioning task metrics](https://mlsanity.com/paper/2207.01733/) (2022)
- [Judge Like Human Examiners: A Weighted Importance Multi-Point Evaluation Framework for Generative Tasks with Long-form Answers](https://mlsanity.com/paper/2604.11246/) (2026)

## Abstract (arXiv)

This paper discusses two existing approaches to the correlation analysis between automatic evaluation metrics and human scores in the area of natural language generation. Our experiments show that depending on the usage of a system- or sentence-level correlation analysis, correlation results between automatic scores and human judgments are inconsistent.