mlsanityData to 7 Oct 2026

Spurious Correlations in Reference-Free Evaluation of Text Generation

arXiv 2204.09890 ↗First posted 21 Apr 2022cs.CLcited by 26

Not on the map. Why?

Abstract (arXiv)

Model-based, reference-free evaluation metrics have been proposed as a fast and cost-effective approach to evaluate Natural Language Generation (NLG) systems. Despite promising recent results, we find evidence that reference-free evaluation metrics of summarization and dialog generation may be relying on spurious correlations with measures such as word overlap, perplexity, and length. We further observe that for text summarization, these metrics have high error rates when ranking current state-of-the-art abstractive summarization systems. We demonstrate that these errors can be mitigated by explicitly designing evaluation metrics to avoid spurious features in reference-free evaluation.

Key facts. Cited by 26 papers in the map (data through 7 Oct 2026). Builds on 41 of its 50 references that are in the map. First posted on arXiv 21 Apr 2022.

Builds on

41 of 50 references.

References · 41

  1. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  2. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  3. 2015Domain-Adversarial Training of Neural Networkscited by 3,023
  4. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  5. 2017Get To The Point: Summarization with Pointer-Generator Networkscited by 1,281
  6. 2019PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarizationcited by 1,100
  7. 2018Personalizing Dialogue Agents: I have a dog, do you have pets too?cited by 820
  8. 2020SummEval: Re-evaluating Summarization Evaluationcited by 666
  9. 2017DailyDialog: A Manually Labelled Multi-turn Dialogue Datasetcited by 649
  10. 2019Evaluating the Factual Consistency of Abstractive Text Summarizationcited by 578
  11. 2018Wizard of Wikipedia: Knowledge-Powered Conversational agentscited by 530
  12. 2020Asking and Answering Questions to Evaluate the Factual Consistency of Summariescited by 376
29 more
  1. 2016How NOT To Evaluate Your Dialogue System: An Empirical Study of Unsupervised Evaluation Metrics for Dialogue Response Generationcited by 356
  2. 2018Newsroom: A Dataset of 1.3 Million Summaries with Diverse Extractive Strategiescited by 292
  3. 2020FEQA: A Question Answering Evaluation Framework for Faithfulness Assessment in Abstractive Summarizationcited by 292
  4. 2021Understanding Factuality in Abstractive Summarization with FRANK: A Benchmark for Factuality Metricscited by 285
  5. 2020Evaluation of Text Generation: A Surveycited by 275
  6. 2021The GEM Benchmark: Natural Language Generation, its Evaluation and Metricscited by 239
  7. 2018Bottom-Up Abstractive Summarizationcited by 177
  8. 2021To Ship or Not to Ship: An Extensive Evaluation of Automatic Metrics for Machine Translationcited by 161
  9. 2018Fast Abstractive Summarization with Reinforce-Selected Sentence Rewritingcited by 158
  10. 2020Tangled up in BLEU: Reevaluating the Evaluation of Automatic Machine Translation Evaluation Metricscited by 155
  11. 2020Re-evaluating Evaluation in Text Summarizationcited by 144
  12. 2017Faithful to the Original: Fact Aware Neural Abstractive Summarizationcited by 141
  13. 2021Annotating and Modeling Fine-grained Factuality in Summarizationcited by 131
  14. 2020Evaluating Factuality in Generation with Dependency-level Entailmentcited by 122
  15. 2021A Comprehensive Assessment of Dialog Evaluation Metricscited by 90
  16. 2020GO FIGURE: A Meta Evaluation of Factuality in Summarizationcited by 75
  17. 2018Multi-Reward Reinforced Summarization with Saliency and Entailmentcited by 73
  18. 2020Dialogue Response Ranking Training with Large-Scale Human Feedback Datacited by 68
  19. 2018Learning to Extract Coherent Summary via Deep Reinforcement Learningcited by 63
  20. 2018Neural Document Summarization by Jointly Learning to Score and Select Sentencescited by 56
  21. 2020Learning an Unreferenced Metric for Online Dialogue Evaluationcited by 52
  22. 2021Faithful or Extractive? On Mitigating the Faithfulness-Abstractiveness Trade-off in Abstractive Summarizationcited by 50
  23. 2018A Unified Model for Extractive and Abstractive Summarization using Inconsistency Losscited by 47
  24. 2020Deconstruct to Reconstruct a Configurable Evaluation Metric for Open-Domain Dialogue Systemscited by 47
  25. 2018BanditSum: Extractive Summarization as a Contextual Banditcited by 46
  26. 2018Improving Abstraction in Text Summarizationcited by 35
  27. 2020Curious Case of Language Generation Evaluation Metrics: A Cautionary Talecited by 28
  28. 2018Soft Layer-Specific Multi-Task Summarization with Entailment and Question Generationcited by 25
  29. 2018Closed-Book Training to Improve Summarization Encoder Memorycited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 6; 2023: 5; 2024: 9; 2025: 4; 2026: 2.

26 papers.

Sort:

By area:

  1. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 865
  2. 2023Benchmarking Large Language Models for News Summarizationcited by 444
  3. 2022News Summarization and Evaluation in the Era of GPT-3cited by 324
  4. 2024RM-Bench: Benchmarking Reward Models of Language Models with Subtlety and Stylecited by 168
  5. 2023LLMs as Narcissistic Evaluators: When Ego Inflates Evaluation Scorescited by 88
  6. 2024Beyond prompt brittleness: Evaluating the reliability and consistency of political worldviews in LLMscited by 55
  7. 2022On the Limitations of Reference-Free Evaluations of Generated Textcited by 52
  8. 2024Explaining Length Bias in LLM-Based Preference Evaluationscited by 48
  9. 2023Opportunities and Risks of LLMs for Scalable Deliberation with Poliscited by 41
  10. 2024Fine-Grained Natural Language Inference Based Faithfulness Evaluation for Diverse Summarisation Taskscited by 23
  11. 2023A Meta-Evaluation of Faithfulness Metrics for Long-Form Hospital-Course Summarizationcited by 14
  12. 2022SESCORE2: Learning Text Generation Evaluation via Synthesizing Realistic Mistakescited by 9
14 more
  1. 2024DESTEIN: Navigating Detoxification of Language Models via Universal Steering Pairs and Head-wise Activation Fusioncited by 9
  2. 2025Reference-free Evaluation Metrics for Text Generation: A Surveycited by 9
  3. 2025Uni-DPO: A Unified Paradigm for Dynamic Preference Optimization of LLMscited by 9
  4. 2023What are the Desired Characteristics of Calibration Sets? Identifying Correlates on Long Form Scientific Summarizationcited by 8
  5. 2025Evaluating Evaluation Metrics -- The Mirage of Hallucination Detectioncited by 6
  6. 2024Mitigating the Impact of Reference Quality on Evaluation of Summarization Systems with Reference-Free Metricscited by 5
  7. 2025Summarizing Speech: A Comprehensive Surveycited by 5
  8. 2022How to Find Strong Summary Coherence Measures? A Toolbox and a Comparative Study for Summary Coherence Measure Evaluationcited by 4
  9. 2022MaskEval: Weighted MLM-Based Evaluation for Text Summarization and Simplificationcited by 3
  10. 2022Social Biases in Automatic Evaluation Metrics for NLGcited by 3
  11. 2024What is Your Favorite Gender, MLM? Gender Bias Evaluation in Multilingual Masked Language Modelscited by 1
  12. 2026Quality Is Not a Safety Proxy Under Quantizationcited by 1
  13. 2024On the Shortcut Learning in Multilingual Neural Machine Translationcited by 0
  14. 2026SP^2DPO: An LLM-assisted Semantic Per-Pair DPO Generalizationcited by 0