# Perception Score, A Learned Metric for Open-ended Text Generation Evaluation

arXiv: https://arxiv.org/abs/2008.03082 · HTML: https://mlsanity.com/paper/2008.03082/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Aug 2020.

## Built on by (most cited first, 7 of 7)

- [CheckEval: A reliable LLM-as-a-Judge framework for evaluating text generation using checklists](https://mlsanity.com/paper/2403.18771/) (2024)
- [Is GPT-3 Text Indistinguishable from Human Text? Scarecrow: A Framework for Scrutinizing Machine Text](https://mlsanity.com/paper/2107.01294/) (2021)
- [SMURF: SeMantic and linguistic UndeRstanding Fusion for Caption Evaluation via Typicality Analysis](https://mlsanity.com/paper/2106.01444/) (2021)
- [Which is better? Exploring Prompting Strategy For LLM-based Metrics](https://mlsanity.com/paper/2311.03754/) (2023)
- [LEGOEval: An Open-Source Toolkit for Dialogue System Evaluation via Crowdsourcing](https://mlsanity.com/paper/2105.01992/) (2021)
- [IntentionQA: A Benchmark for Evaluating Purchase Intention Comprehension Abilities of Language Models in E-commerce](https://mlsanity.com/paper/2406.10173/) (2024)
- [Med-CoDE: Medical Critique based Disagreement Evaluation Framework](https://mlsanity.com/paper/2504.15330/) (2025)

## Abstract (arXiv)

Automatic evaluation for open-ended natural language generation tasks remains a challenge. Existing metrics such as BLEU show a low correlation with human judgment. We propose a novel and powerful learning-based evaluation metric: Perception Score. The method measures the overall quality of the generation and scores holistically instead of only focusing on one evaluation criteria, such as word overlapping. Moreover, it also shows the amount of uncertainty about its evaluation result. By connecting the uncertainty, Perception Score gives a more accurate evaluation for the generation system. Perception Score provides state-of-the-art results on two conditional generation tasks and two unconditional generation tasks.