mlsanityData to 7 Oct 2026

Is ChatGPT a Good NLG Evaluator? A Preliminary Study

arXiv 2303.04048 ↗First posted 7 Mar 2023cs.CL cs.AIcited by 420

Not on the map. Why?

Cited by 36 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Recently, the emergence of ChatGPT has attracted wide attention from the computational linguistics community. Many prior studies have shown that ChatGPT achieves remarkable performance on various NLP tasks in terms of automatic evaluation metrics. However, the ability of ChatGPT to serve as an evaluation metric is still underexplored. Considering assessing the quality of natural language generation (NLG) models is an arduous task and NLG metrics notoriously show their poor correlation with human judgments, we wonder whether ChatGPT is a good NLG evaluation metric. In this report, we provide a preliminary meta-evaluation on ChatGPT to show its reliability as an NLG metric. In detail, we regard ChatGPT as a human evaluator and give task-specific (e.g., summarization) and aspect-specific (e.g., relevance) instruction to prompt ChatGPT to evaluate the generated results of NLG models. We conduct experiments on five NLG meta-evaluation datasets (including summarization, story generation and data-to-text tasks). Experimental results show that compared with previous automatic metrics, ChatGPT achieves state-of-the-art or competitive correlation with human judgments in most cases. In addition, we find that the effectiveness of the ChatGPT evaluator might be influenced by the creation method of the meta-evaluation datasets. For the meta-evaluation datasets which are created greatly depending on the reference and thus are biased, the ChatGPT evaluator might lose its effectiveness. We hope our preliminary study could prompt the emergence of a general-purposed reliable NLG metric.

Key facts. Cited by 420 papers in the map (data through 7 Oct 2026). Builds on 33 of its 51 references that are in the map. First posted on arXiv 7 Mar 2023.

Builds on

33 of 51 references.

References · 33

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  4. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  5. 2020BLEURT: Learning Robust Metrics for Text Generationcited by 1,196
  6. 2015A Diversity-Promoting Objective Function for Neural Conversation Modelscited by 1,068
  7. 2015Teaching Machines to Read and Comprehendcited by 1,039
  8. 2020COMET: A Neural Framework for MT Evaluationcited by 1,024
  9. 2023A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivitycited by 888
  10. 2021BARTScore: Evaluating Generated Text as Text Generationcited by 719
  11. 2020SummEval: Re-evaluating Summarization Evaluationcited by 666
  12. 2023Is ChatGPT a General-Purpose Natural Language Processing Task Solver?cited by 486
21 more
  1. 2023Large Language Models Are State-of-the-Art Evaluators of Translation Qualitycited by 416
  2. 2019MoverScore: Text Generation Evaluating with Contextualized Embeddings and Earth Mover Distancecited by 415
  3. 2023GPTScore: Evaluate as You Desirecited by 357
  4. 2018Newsroom: A Dataset of 1.3 Million Summaries with Diverse Extractive Strategiescited by 292
  5. 2023Mathematical Capabilities of ChatGPTcited by 263
  6. 2023ChatIE: Zero-Shot Information Extraction via Chatting with ChatGPTcited by 250
  7. 2023ChatGPT: Jack of all trades, master of nonecited by 204
  8. 2023Is ChatGPT better than Human Annotators? Potential and Limitations of ChatGPT in Explaining Implicit Hate Speechcited by 165
  9. 2023Can ChatGPT Understand Too? A Comparative Study on ChatGPT and Fine-tuned BERTcited by 155
  10. 2020Re-evaluating Evaluation in Text Summarizationcited by 144
  11. 2023Exploring the Limits of ChatGPT for Query or Aspect-based Text Summarizationcited by 122
  12. 2019Handling Divergent Reference Texts when Evaluating Table-to-Text Generationcited by 119
  13. 2020Automatic Machine Translation Evaluation in Many Languages via Zero-Shot Paraphrasingcited by 114
  14. 2022ChatGPT Makes Medicine Easy to Swallow: An Exploratory Case Study on Simplified Radiology Reportscited by 100
  15. 2023Can ChatGPT Assess Human Personalities? A General Evaluation Frameworkcited by 69
  16. 2021OpenMEVA: A Benchmark for Evaluating Open-ended Story Generation Metricscited by 61
  17. 2023Can ChatGPT Write a Good Boolean Query for Systematic Review Literature Search?cited by 54
  18. 2022MAUVE Scores for Generative Models: Theory and Practicecited by 34
  19. 2022StoryER: Automatic Story Evaluation via Ranking, Rating and Reasoningcited by 21
  20. 2023Zero-Shot Cross-Lingual Summarization via Large Language Modelscited by 16
  21. 2023Dr ChatGPT, tell me what I want to hear: How prompt knowledge impacts health answer correctnesscited by 15

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 112; 2024: 173; 2025: 91; 2026: 44.

420 papers.

Sort:

By area:

  1. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,205
  2. 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,582
  3. 2024From Local to Global: A Graph RAG Approach to Query-Focused Summarizationcited by 1,411
  4. 2023A Survey on Evaluation of Large Language Modelscited by 1,227
  5. 2023Can Large Language Models Be an Alternative to Human Evaluations?cited by 729
  6. 2023ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debatecited by 724
  7. 2023GPTFUZZER: Red Teaming Large Language Models with Auto-Generated Jailbreak Promptscited by 520
  8. 2024From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judgecited by 480
  9. 2023Harnessing the Power of LLMs in Practice: A Survey on ChatGPT and Beyondcited by 409
  10. 2023Ragas: Automated Evaluation of Retrieval Augmented Generationcited by 408
  11. 2023Learning to Compress Prompts with Gist Tokenscited by 379
  12. 2024Justice or Prejudice? Quantifying Biases in LLM-as-a-Judgecited by 354
38 more
  1. 2023Evaluating Large Language Models at Evaluating Instruction Followingcited by 290
  2. 2023ARES: An Automated Evaluation Framework for Retrieval-Augmented Generation Systemscited by 259
  3. 2024LLMs instead of Human Judges? A Large Scale Empirical Study across 20 NLP Evaluation Taskscited by 236
  4. 2023Driving with LLMs: Fusing Object-Level Vector Modality for Explainable Autonomous Drivingcited by 231
  5. 2024Leak, Cheat, Repeat: Data Contamination and Evaluation Malpractices in Closed-Source LLMscited by 224
  6. 2023ChatGPT Beyond English: Towards a Comprehensive Evaluation of Large Language Models in Multilingual Learningcited by 203
  7. 2023Evaluating Large Language Models: A Comprehensive Surveycited by 200
  8. 2023Universal Self-Consistency for Large Language Model Generationcited by 198
  9. 2023Benchmarking Foundation Models with Language-Model-as-an-Examinercited by 172
  10. 2023Large Language Model Routing with Benchmark Datasetscited by 166
  11. 2023Summary of ChatGPT-Related Research and Perspective Towards the Future of Large Language Modelscited by 152
  12. 2023Benchmarking Cognitive Biases in Large Language Models as Evaluatorscited by 147
  13. 2024Aligning with Human Judgement: The Role of Pairwise Preference in Large Language Model Evaluatorscited by 146
  14. 2024Large Language Models are Inconsistent and Biased Evaluatorscited by 146
  15. 2023BooookScore: A systematic exploration of book-length summarization in the era of LLMscited by 137
  16. 2024InfiAgent-DABench: Evaluating Agents on Data Analysis Taskscited by 134
  17. 2024InFoBench: Evaluating Instruction Following Ability in Large Language Modelscited by 128
  18. 2023Human-like Summarization Evaluation with ChatGPTcited by 126
  19. 2023A Systematic Study and Comprehensive Evaluation of ChatGPT on Benchmark Datasetscited by 123
  20. 2025Process Reward Models That Thinkcited by 118
  21. 2024Is LLM-as-a-Judge Robust? Investigating Universal Adversarial Attacks on Zero-shot LLM Assessmentcited by 117
  22. 2023Artificial Artificial Artificial Intelligence: Crowd Workers Widely Use Large Language Models for Text Production Taskscited by 102
  23. 2023CoAnnotating: Uncertainty-Guided Work Allocation between Human and Large Language Models for Data Annotationcited by 93
  24. 2023Wider and Deeper LLM Networks are Fairer LLM Evaluatorscited by 91
  25. 2023Jatmo: Prompt Injection Defense by Task-Specific Finetuningcited by 89
  26. 2023Can ChatGPT Reproduce Human-Generated Labels? A Study of Social Computing Taskscited by 88
  27. 2024CPsyCoun: A Report-based Multi-turn Dialogue Reconstruction and Evaluation Framework for Chinese Psychological Counselingcited by 88
  28. 2023Are Large Language Model-based Evaluators the Solution to Scaling Up Multilingual Evaluation?cited by 87
  29. 2023AgentSims: An Open-Source Sandbox for Large Language Model Evaluationcited by 86
  30. 2023Exploring the Use of Large Language Models for Reference-Free Text Quality Evaluation: An Empirical Studycited by 85
  31. 2024FineSurE: Fine-grained Summarization Evaluation using LLMscited by 83
  32. 2023Safety Assessment of Chinese Large Language Modelscited by 82
  33. 2023TrueTeacher: Learning Factual Consistency Evaluation with Large Language Modelscited by 82
  34. 2023On Learning to Summarize with Large Language Models as Referencescited by 80
  35. 2023ChatGPT as a Factual Inconsistency Evaluator for Text Summarizationcited by 75
  36. 2023GPTEval: A Survey on Assessments of ChatGPT and GPT-4cited by 73
  37. 2024Foundational Autoraters: Taming Large Language Models for Better Automatic Evaluationcited by 72
  38. 2024LLM-based NLG Evaluation: Current Status and Challengescited by 71

50 most cited of 420 All 420 (JSON)