# Referenceless Quality Estimation for Natural Language Generation

arXiv: https://arxiv.org/abs/1708.01759 · HTML: https://mlsanity.com/paper/1708.01759/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 5 Aug 2017.

## Built on by (most cited first, 4 of 4)

- [Evaluating the Morphosyntactic Well-formedness of Generated Texts](https://mlsanity.com/paper/2103.16590/) (2021)
- [Naturalness Evaluation of Natural Language Generation in Task-oriented Dialogues using BERT](https://mlsanity.com/paper/2109.02938/) (2021)
- [How to Evaluate Your Dialogue Models: A Review of Approaches](https://mlsanity.com/paper/2108.01369/) (2021)
- [Clinical Reading Comprehension with Encoder-Decoder Models Enhanced by Direct Preference Optimization](https://mlsanity.com/paper/2407.14000/) (2024)

## Abstract (arXiv)

Traditional automatic evaluation measures for natural language generation (NLG) use costly human-authored references to estimate the quality of a system output. In this paper, we propose a referenceless quality estimation (QE) approach based on recurrent neural networks, which predicts a quality score for a NLG system output by comparing it to the source meaning representation only. Our method outperforms traditional metrics and a constant baseline in most respects; we also show that synthetic data helps to increase correlation results by 21% compared to the base system. Our results are comparable to results obtained in similar QE tasks despite the more challenging setting.