mlsanityData to 14 Oct 2026

On the Limitations of Reference-Free Evaluations of Generated Text

arXiv 2210.12563 ↗First posted 22 Oct 2022cs.CLcited by 52

Not on the map. Why?

Abstract (arXiv)

There is significant interest in developing evaluation metrics which accurately estimate the quality of generated text without the aid of a human-written reference text, which can be time consuming and expensive to collect or entirely unavailable in online applications. However, in this work, we demonstrate that these reference-free metrics are inherently biased and limited in their ability to evaluate generated text, and we argue that they should not be used to measure progress on tasks like machine translation or summarization. We show how reference-free metrics are equivalent to using one generation model to evaluate another, which has several limitations: (1) the metrics can be optimized at test time to find the approximate best-possible output, (2) they are inherently biased toward models which are more similar to their own, and (3) they can be biased against higher-quality outputs, including those written by humans. Therefore, we recommend that reference-free metrics should be used as diagnostic tools for analyzing and understanding model behavior instead of measures of how well models perform a task, in which the goal is to achieve as high of a score as possible.

Key facts. Cited by 52 papers in the map (data through 14 Oct 2026). Builds on 32 of its 47 references that are in the map. First posted on arXiv 22 Oct 2022.

Builds on

32 of 47 references.

References · 32

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  3. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  4. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  5. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,424
  6. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,799
  7. 2020BLEURT: Learning Robust Metrics for Text Generationcited by 1,203
  8. 2020COMET: A Neural Framework for MT Evaluationcited by 1,033
  9. 2016Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyondcited by 812
  10. 2020SummEval: Re-evaluating Summarization Evaluationcited by 674
  11. 2016SummaRuNNer: A Recurrent Neural Network based Sequence Model for Extractive Summarization of Documentscited by 246
  12. 2021QuestEval: Summarization Asks for Fact-based Evaluationcited by 244
20 more
  1. 2020USR: An Unsupervised and Reference Free Evaluation Metric for Dialog Generationcited by 188
  2. 2019Facebook FAIR's WMT19 News Translation Task Submissioncited by 179
  3. 2020Re-evaluating Evaluation in Text Summarizationcited by 145
  4. 2021Q2: Evaluating Factual Consistency in Knowledge-Grounded Dialogues via Question Generation and Question Answeringcited by 132
  5. 2020Automatic Machine Translation Evaluation in Many Languages via Zero-Shot Paraphrasingcited by 115
  6. 2021High Quality Rather than High Model Probability: Minimum Bayes Risk Decoding with Neural Metricscited by 104
  7. 2019Answers Unite! Unsupervised Metrics for Reinforced Summarization Modelscited by 92
  8. 2020Towards Question-Answering as an Automatic Metric for Evaluating the Content Quality of a Summarycited by 88
  9. 2020Fill in the BLANC: Human-free quality estimation of document summariescited by 82
  10. 2021A Statistical Analysis of Summarization Evaluation Metrics using Resampling Methodscited by 72
  11. 2020SUPERT: Towards New Frontiers in Unsupervised Evaluation Metrics for Multi-Document Summarizationcited by 69
  12. 2022Quality-Aware Decoding for Neural Machine Translationcited by 49
  13. 2021Finding a Balanced Degree of Automation for Summary Evaluationcited by 43
  14. 2022Spurious Correlations in Reference-Free Evaluation of Text Generationcited by 26
  15. 2020SacreROUGE: An Open-Source Library for Using and Developing Summarization Evaluation Metricscited by 18
  16. 2019SumQE: a BERT-based Summary Quality Estimation Modelcited by 17
  17. 2019Reference-less Quality Estimation of Text Simplification Systemscited by 10
  18. 2021Assessing Reference-Free Peer Evaluation for Machine Translationcited by 8
  19. 2020Simple-QE: Better Automatic Quality Estimation for Text Simplificationcited by 5
  20. 2022Repro: An Open-Source Library for Improving the Reproducibility and Usability of Publicly Available Research Codecited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 1; 2023: 9; 2024: 18; 2025: 17; 2026: 7.

52 papers.

Sort:

By area:

  1. 2024Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothingcited by 346
  2. 2024Self-Preference Bias in LLM-as-a-Judgecited by 258
  3. 2024Large Language Models are Inconsistent and Biased Evaluatorscited by 153
  4. 2022Revisiting the Gold Standard: Grounding Summarization Evaluation with Robust Human Evaluationcited by 142
  5. 2023LLMs as Narcissistic Evaluators: When Ego Inflates Evaluation Scorescited by 90
  6. 2025No Free Labels: Limitations of LLM-as-a-Judge Without Human Groundingcited by 54
  7. 2023What Comes Next? Evaluating Uncertainty in Neural Text Generators Against Human Production Variabilitycited by 51
  8. 2025Play Favorites: A Statistical Method to Measure Self-Bias in LLM-as-a-Judgecited by 36
  9. 2024Do Language Models Enjoy Their Own Stories? Prompting Large Language Models for Automatic Story Evaluationcited by 32
  10. 2025MINERVA: Evaluating Complex Video Reasoningcited by 32
  11. 2024Recent Advances in Generative AI and Large Language Models: Current Status, Challenges, and Perspectivescited by 30
  12. 2023Less is More for Long Document Summary Evaluation by LLMscited by 28
38 more
  1. 2024ACLSum: A New Dataset for Aspect-based Summarization of Scientific Publicationscited by 20
  2. 2024RevisEval: Improving LLM-as-a-Judge via Response-Adapted Referencescited by 20
  3. 2023SEMQA: Semi-Extractive Multi-Source Question Answeringcited by 19
  4. 2023Aligning Neural Machine Translation Models: Human Feedback in Training and Inferencecited by 19
  5. 2024Know When To Stop: A Study of Semantic Drift in Text Generationcited by 18
  6. 2023Walking a Tightrope -- Evaluating Large Language Models in High-Risk Domainscited by 15
  7. 2026CARE: Confounder-Aware Aggregation for Reliable LLM Evaluationcited by 13
  8. 2025Reference-free Evaluation Metrics for Text Generation: A Surveycited by 9
  9. 2024QAPyramid: Fine-grained Evaluation of Content Selection for Text Summarizationcited by 8
  10. 2023RADE: Reference-Assisted Dialogue Evaluation for Open-Domain Dialoguecited by 7
  11. 2024textttCOSMIC: Mutual Information for Task-Agnostic Summarization Evaluationcited by 6
  12. 2024Is Reference Necessary in the Evaluation of NLG Systems? When and Where?cited by 6
  13. 2025Document Summarization with Conformal Importance Guaranteescited by 6
  14. 2023CLEME: Debiasing Multi-reference Evaluation for Grammatical Error Correctioncited by 5
  15. 2024Context-Aware Assistant Selection for Improved Inference Acceleration with Large Language Modelscited by 5
  16. 2024Audio Description Generation in the Era of LLMs and VLMs: A Review of Transferable Generative AI Technologiescited by 5
  17. 2025What's the Difference? Supporting Users in Identifying the Effects of Prompt and Model Changes Through Token Patternscited by 4
  18. 2025M2PO: Multi-Perspective Multi-Pair Preference Optimization for Machine Translationcited by 4
  19. 2024REFeREE: A REference-FREE Model-Based Metric for Text Simplificationcited by 3
  20. 2024Exploring the Limits of Fine-grained LLM-based Physics Inference via Premise Removal Interventionscited by 3
  21. 2023Interpretation modeling: Social grounding of sentences by reasoning over their implicit moral judgmentscited by 2
  22. 2024A Critical Look at Meta-evaluating Summarisation Evaluation Metricscited by 2
  23. 2025Cascaded Self-Evaluation Augmented Training for Lightweight Multimodal LLMscited by 2
  24. 2025Taming LLMs with Negative Samples: A Reference-Free Framework to Evaluate Presentation Content with Actionable Feedbackcited by 2
  25. 2024Surveying the Landscape of Image Captioning Evaluation: A Comprehensive Taxonomy, Trends and Metrics Analysiscited by 1
  26. 2025HalluCounter: Reference-free LLM Hallucination Detection in the Wild!cited by 1
  27. 2025ContrastScore: Towards Higher Quality, Less Biased, More Efficient Evaluation Metrics with Contrastive Evaluationcited by 1
  28. 2025PoSh: Using Scene Graphs To Guide LLMs-as-a-Judge For Detailed Image Descriptionscited by 1
  29. 2025Music Recommendation with Large Language Models: Challenges, Opportunities, and Evaluationcited by 1
  30. 2026MINERVA-Cultural: A Benchmark for Cultural and Multilingual Long Video Reasoningcited by 1
  31. 2026HoWToBench: Holistic Evaluation for LLM's Capability in Human-level Writing using Tree of Writingcited by 1
  32. 2026MATCHA: Matching Text via Contrastive Semantic Alignmentcited by 1
  33. 2026SEFORA: Student Essays with Feedback Corpus and LLM Feedback Evaluation Frameworkcited by 1
  34. 2024Rank It, Then Ask It: Input Reranking for Maximizing the Performance of LLMs on Symmetric Taskscited by 0
  35. 2025GlotEval: A Test Suite for Massively Multilingual Evaluation of Large Language Modelscited by 0
  36. 2025Simple and Effective Baselines for Code Summarisation Evaluationcited by 0
  37. 2025A Conceptual Framework for AI Capability Evaluationscited by 0
  38. 2025TextMineX: Data, Evaluation Framework and Ontology-guided LLM Pipeline for Humanitarian Mine Actioncited by 0

50 most cited of 52 All 52 (JSON)