# NLG Evaluation: Past, Present, Future

arXiv: https://arxiv.org/abs/2605.23715 · HTML: https://mlsanity.com/paper/2605.23715/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 41 references that are in the map.
- First posted on arXiv 22 May 2026.

## Builds on (most cited first, 9 of 9)

- [Beyond Accuracy: Behavioral Testing of NLP models with CheckList](https://mlsanity.com/paper/2005.04118/) (2020)
- [Leak, Cheat, Repeat: Data Contamination and Evaluation Malpractices in Closed-Source LLMs](https://mlsanity.com/paper/2402.03927/) (2024)
- [Tangled up in BLEU: Reevaluating the Evaluation of Automatic Machine Translation Evaluation Metrics](https://mlsanity.com/paper/2006.06264/) (2020)
- [Measuring what Matters: Construct Validity in Large Language Model Benchmarks](https://mlsanity.com/paper/2511.04703/) (2025)
- [MIRAGE: The Illusion of Visual Understanding](https://mlsanity.com/paper/2603.21687/) (2026)
- [Evaluating Semantic Accuracy of Data-to-Text Generation with Natural Language Inference](https://mlsanity.com/paper/2011.10819/) (2020)
- [A Gold Standard Methodology for Evaluating Accuracy in Data-To-Text Systems](https://mlsanity.com/paper/2011.03992/) (2020)
- [Deconstructing NLG Evaluation: Evaluation Practices, Assumptions, and Their Implications](https://mlsanity.com/paper/2205.06828/) (2022)
- [We Should Evaluate Real-World Impact](https://mlsanity.com/paper/2507.05973/) (2025)

## Built on by (most cited first, 2 of 2)

- [LLM Judges Can Be Too Generous When There Is No Reference Answer](https://mlsanity.com/paper/2607.12885/) (2026)
- [EuroExec: Frontier Language Models Fall Short of Expert Judgment on European Executive Decision Tasks](https://mlsanity.com/paper/2608.04549/) (2026)

## Abstract (arXiv)

Natural Language Generation (NLG) evaluation has changed dramatically since 1990, and will continue to evolve in the future. In 1990, when NLG had close ties to linguistics, there was very little formal experimental evaluation in the modern sense. In 2026, when NLG is closely linked to machine learning, experimental evaluation is expected and indeed fundamental to research. Many evaluation techniques were developed over this period, including most recently LLM-as-Judge. I expect NLG evaluation will continue to evolve in the future. In particular, impact, qualitative, and safety evaluation will become more important as large numbers of people routinely use NLG technology.