mlsanityData to 7 Oct 2026

A Dual-Perspective NLG Meta-Evaluation Framework with Automatic Benchmark and Better Interpretability

arXiv 2502.12052 ↗First posted 17 Feb 2025cs.CLcited by 1

Not on the map. Why?

Abstract (arXiv)

In NLG meta-evaluation, evaluation metrics are typically assessed based on their consistency with humans. However, we identify some limitations in traditional NLG meta-evaluation approaches, such as issues in handling human ratings and ambiguous selections of correlation measures, which undermine the effectiveness of meta-evaluation. In this work, we propose a dual-perspective NLG meta-evaluation framework that focuses on different evaluation capabilities, thereby providing better interpretability. In addition, we introduce a method of automatically constructing the corresponding benchmarks without requiring new human annotations. Furthermore, we conduct experiments with 16 representative LLMs as the evaluators based on our proposed framework, comprehensively analyzing their evaluation performance from different perspectives.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 45 of its 64 references that are in the map. First posted on arXiv 17 Feb 2025.

Builds on

45 of 64 references.

References · 45

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  4. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,205
  5. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,928
  6. 2023Large Language Models are not Fair Evaluatorscited by 1,050
  7. 2023Can Large Language Models Be an Alternative to Human Evaluations?cited by 729
  8. 2021BARTScore: Evaluating Generated Text as Text Generationcited by 719
  9. 2024Phi-4 Technical Reportcited by 675
  10. 2020SummEval: Re-evaluating Summarization Evaluationcited by 666
  11. 2023Prometheus: Inducing Fine-grained Evaluation Capability in Language Modelscited by 558
  12. 2024Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Modelscited by 431
33 more
  1. 2023Is ChatGPT a Good NLG Evaluator? A Preliminary Studycited by 420
  2. 2024MLLM-as-a-Judge: Assessing Multimodal LLM-as-a-Judge with Vision-Language Benchmarkcited by 362
  3. 2024Justice or Prejudice? Quantifying Biases in LLM-as-a-Judgecited by 354
  4. 2024JudgeBench: A Benchmark for Evaluating LLM-based Judgescited by 294
  5. 2023Evaluating Large Language Models at Evaluating Instruction Followingcited by 290
  6. 2024LLMs instead of Human Judges? A Large Scale Empirical Study across 20 NLP Evaluation Taskscited by 236
  7. 2023FLASK: Fine-grained Language Model Evaluation based on Alignment Skill Setscited by 198
  8. 2024Judging the Judges: Evaluating Alignment and Vulnerabilities in LLMs-as-Judgescited by 197
  9. 2020USR: An Unsupervised and Reference Free Evaluation Metric for Dialog Generationcited by 186
  10. 2023Generative Judge for Evaluating Alignmentcited by 163
  11. 2024Aligning with Human Judgement: The Role of Pairwise Preference in Large Language Model Evaluatorscited by 146
  12. 2023Wider and Deeper LLM Networks are Fairer LLM Evaluatorscited by 91
  13. 2024The BiGGen Bench: A Principled Benchmark for Fine-grained Evaluation of Language Models with Language Modelscited by 89
  14. 2023Benchmarking Generation and Evaluation Capabilities of Large Language Models for Instruction Controllable Summarizationcited by 69
  15. 2023CritiqueLLM: Towards an Informative Critique Generation Model for Evaluation of Large Language Model Generationcited by 65
  16. 2021Perturbation CheckLists for Evaluating NLG Evaluation Metricscited by 49
  17. 2022On the Blind Spots of Model-Based Evaluation Metrics for Text Generationcited by 48
  18. 2024CompassJudger-1: All-in-one Judge Model Helps Model Evaluation and Evolutioncited by 42
  19. 2024Are LLM-based Evaluators Confusing NLG Quality Criteria?cited by 35
  20. 2024HD-Eval: Aligning Large Language Model Evaluators Through Hierarchical Criteria Decompositioncited by 33
  21. 2024Do Language Models Enjoy Their Own Stories? Prompting Large Language Models for Automatic Story Evaluationcited by 32
  22. 2024Evaluating the Consistency of LLM Evaluatorscited by 24
  23. 2024Guardians of the Machine Translation Meta-Evaluation: Sentinel Metrics Fall In!cited by 21
  24. 2024Measuring the Inconsistency of Large Language Models in Preferential Rankingcited by 18
  25. 2024One Prompt To Rule Them All: LLMs for Opinion Summary Evaluationcited by 16
  26. 2024Can Automatic Metrics Assess High-Quality Translations?cited by 16
  27. 2023xDial-Eval: A Multilingual Open-Domain Dialogue Evaluation Benchmarkcited by 13
  28. 2024DnA-Eval: Enhancing Large Language Model Evaluation through Decomposition and Aggregationcited by 12
  29. 2024DHP Benchmark: Are LLMs Good NLG Evaluators?cited by 12
  30. 2024Soda-Eval: Open-Domain Dialogue Evaluation in the age of LLMscited by 10
  31. 2024Beyond Correlation: Interpretable Evaluation of Machine Translation Metricscited by 10
  32. 2020An Effectiveness Metric for Ordinal Classification: Formal Properties and Experimental Resultscited by 7
  33. 2024Analyzing and Evaluating Correlation Measures in NLG Meta-Evaluationcited by 4

Built on by

Cited by year

Citing papers per year: 2025: 1.

1 paper.

  1. 2025Zero-shot Benchmarking: A Framework for Flexible and Scalable Automatic Evaluation of Language Modelscited by 14