mlsanityData to 7 Oct 2026

G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment

arXiv 2303.16634 ↗First posted 29 Mar 2023cs.CL cs.AIcited by 2,205

Not on the map. Why?

Cited by 309 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

The quality of texts generated by natural language generation (NLG) systems is hard to measure automatically. Conventional reference-based metrics, such as BLEU and ROUGE, have been shown to have relatively low correlation with human judgments, especially for tasks that require creativity and diversity. Recent studies suggest using large language models (LLMs) as reference-free metrics for NLG evaluation, which have the benefit of being applicable to new tasks that lack human references. However, these LLM-based evaluators still have lower human correspondence than medium-size neural evaluators. In this work, we present G-Eval, a framework of using large language models with chain-of-thoughts (CoT) and a form-filling paradigm, to assess the quality of NLG outputs. We experiment with two generation tasks, text summarization and dialogue generation. We show that G-Eval with GPT-4 as the backbone model achieves a Spearman correlation of 0.514 with human on summarization task, outperforming all previous methods by a large margin. We also propose preliminary analysis on the behavior of LLM-based evaluators, and highlight the potential issue of LLM-based evaluators having a bias towards the LLM-generated texts. The code is at https://github.com/nlpyang/geval

Key facts. Cited by 2,205 papers in the map (data through 7 Oct 2026). Builds on 27 of its 40 references that are in the map. First posted on arXiv 29 Mar 2023.

Builds on

27 of 40 references.

References · 27

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  5. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  6. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  7. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,308
  8. 2015Teaching Machines to Read and Comprehendcited by 1,039
  9. 2021BARTScore: Evaluating Generated Text as Text Generationcited by 719
  10. 2020SummEval: Re-evaluating Summarization Evaluationcited by 666
  11. 2019Evaluating the Factual Consistency of Abstractive Text Summarizationcited by 578
  12. 2023Benchmarking Large Language Models for News Summarizationcited by 444
15 more
  1. 2023Is ChatGPT a Good NLG Evaluator? A Preliminary Studycited by 420
  2. 2023Large Language Models Are State-of-the-Art Evaluators of Translation Qualitycited by 416
  3. 2019MoverScore: Text Generation Evaluating with Contextualized Embeddings and Earth Mover Distancecited by 415
  4. 2020Asking and Answering Questions to Evaluate the Factual Consistency of Summariescited by 376
  5. 2023GPTScore: Evaluate as You Desirecited by 357
  6. 2020FEQA: A Question Answering Evaluation Framework for Faithfulness Assessment in Abstractive Summarizationcited by 292
  7. 2022Towards a Unified Multi-Dimensional Evaluator for Text Generationcited by 264
  8. 2020USR: An Unsupervised and Reference Free Evaluation Metric for Dialog Generationcited by 186
  9. 2017Faithful to the Original: Fact Aware Neural Abstractive Summarizationcited by 141
  10. 2020Factual Error Correction for Abstractive Summarization Modelscited by 108
  11. 2023Error Analysis Prompting Enables Human-Like Translation Evaluation in Large Language Modelscited by 87
  12. 2019Evaluating Coherence in Dialogue Systems using Entailmentcited by 53
  13. 2019Better Automatic Evaluation of Open-Domain Dialogue Systems with Contextualized Embeddingscited by 52
  14. 2021Bidimensional Leaderboards: Generate and Evaluate Language Hand in Handcited by 32
  15. 2021Towards Quantifiable Dialogue Coherence Evaluationcited by 21

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 217; 2024: 487; 2025: 573; 2026: 928.

2,205 papers.

Sort:

By area:

  1. 2023Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflectioncited by 1,800
  2. 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Daycited by 1,287
  3. 2023FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generationcited by 1,259
  4. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,221
  5. 2023Instruction-Following Evaluation for Large Language Modelscited by 1,143
  6. 2023CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquingcited by 751
  7. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 738
  8. 2023ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debatecited by 724
  9. 2024LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memorycited by 656
  10. 2023ChemCrow: Augmenting large-language models with chemistry toolscited by 609
  11. 2023Prometheus: Inducing Fine-grained Evaluation Capability in Language Modelscited by 558
  12. 2023GPTFUZZER: Red Teaming Large Language Models with Auto-Generated Jailbreak Promptscited by 520
38 more
  1. 2024From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judgecited by 480
  2. 2024LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methodscited by 451
  3. 2024AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agentscited by 450
  4. 2023Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuningcited by 439
  5. 2023CMMLU: Measuring massive multitask language understanding in Chinesecited by 436
  6. 2024Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Modelscited by 431
  7. 2023Harnessing the Power of LLMs in Practice: A Survey on ChatGPT and Beyondcited by 409
  8. 2023Understanding the Effects of RLHF on LLM Generalisation and Diversitycited by 386
  9. 2023DoLa: Decoding by Contrasting Layers Improves Factuality in Large Language Modelscited by 367
  10. 2024Hybrid LLM: Cost-Efficient and Quality-Aware Query Routingcited by 354
  11. 2023Evaluating Large Language Models at Evaluating Instruction Followingcited by 290
  12. 2023AlignScore: Evaluating Factual Consistency with a Unified Alignment Functioncited by 284
  13. 2023LLaVAR: Enhanced Visual Instruction Tuning for Text-Rich Image Understandingcited by 272
  14. 2023The Unlocking Spell on Base LLMs: Rethinking Alignment via In-Context Learningcited by 251
  15. 2023Large Language Model Alignment: A Surveycited by 241
  16. 2024LogicVista: Multimodal LLM Logical Reasoning Benchmark in Visual Contextscited by 240
  17. 2024LLMs instead of Human Judges? A Large Scale Empirical Study across 20 NLP Evaluation Taskscited by 236
  18. 2023Driving with LLMs: Fusing Object-Level Vector Modality for Explainable Autonomous Drivingcited by 231
  19. 2024When Can LLMs Actually Correct Their Own Mistakes? A Critical Survey of Self-Correction of LLMscited by 230
  20. 2023PersonaLLM: Investigating the Ability of Large Language Models to Express Personality Traitscited by 229
  21. 2024Leak, Cheat, Repeat: Data Contamination and Evaluation Malpractices in Closed-Source LLMscited by 224
  22. 2024Agent-as-a-Judge: Evaluate Agents with Agentscited by 223
  23. 2025A Survey on Test-Time Scaling in Large Language Models: What, How, Where, and How Well?cited by 215
  24. 2023Evaluating Large Language Models: A Comprehensive Surveycited by 200
  25. 2024ResearchAgent: Iterative Research Idea Generation over Scientific Literature with Large Language Modelscited by 200
  26. 2023FLASK: Fine-grained Language Model Evaluation based on Alignment Skill Setscited by 198
  27. 2023Universal Self-Consistency for Large Language Model Generationcited by 198
  28. 2023Honeybee: Locality-enhanced Projector for Multimodal LLMcited by 195
  29. 2023OctoPack: Instruction Tuning Code Large Language Modelscited by 178
  30. 2023Benchmarking Foundation Models with Language-Model-as-an-Examinercited by 172
  31. 2024Lessons from the Trenches on Reproducible Evaluation of Language Modelscited by 172
  32. 2025Which Agent Causes Task Failures and When? On Automated Failure Attribution of LLM Multi-Agent Systemscited by 170
  33. 2023Art or Artifice? Large Language Models and the False Promise of Creativitycited by 168
  34. 2023Large Language Model Routing with Benchmark Datasetscited by 166
  35. 2023Generative Judge for Evaluating Alignmentcited by 163
  36. 2023UniChart: A Universal Vision-language Pretrained Model for Chart Comprehension and Reasoningcited by 158
  37. 2023Evaluating Correctness and Faithfulness of Instruction-Following Models for Question Answeringcited by 150
  38. 2023Knowledge Graph Prompting for Multi-Document Question Answeringcited by 150

50 most cited of 2,205 All 2,205 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024A Systematic Evaluation of Large Language Models for Natural Language Generation Tasks
  2. 2024Sample-Efficient Human Evaluation of Large Language Models via Maximum Discrepancy Competition
  3. 2023L-Eval: Instituting Standardized Evaluation for Long Context Language Models
  4. 2023TencentLLMEval: A Hierarchical Evaluation of Real-World Capabilities for Human-Aligned LLMs
  5. 2024ConSiDERS-The-Human Evaluation Framework: Rethinking Human Evaluation for Generative Large Language Models
  6. 2025From Understanding to Generation: An Efficient Shortcut for Evaluating Language Models
  7. 2022Training Language Models with Language Feedback
  8. 2024Can Large Language Models be Trusted for Evaluation? Scalable Meta-Evaluation of LLMs as Evaluators via Agent Debate