mlsanityData to 7 Oct 2026

GPTScore: Evaluate as You Desire

arXiv 2302.04166 ↗First posted 8 Feb 2023cs.CLcited by 357

Not on the map. Why?

Cited by 50 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Generative Artificial Intelligence (AI) has enabled the development of sophisticated models that are capable of producing high-caliber text, images, and other outputs through the utilization of large pre-trained models. Nevertheless, assessing the quality of the generation is an even more arduous task than the generation itself, and this issue has not been given adequate consideration recently. This paper proposes a novel evaluation framework, GPTScore, which utilizes the emergent abilities (e.g., zero-shot instruction) of generative pre-trained models to score generated texts. There are 19 pre-trained models explored in this paper, ranging in size from 80M (e.g., FLAN-T5-small) to 175B (e.g., GPT3). Experimental results on four text generation tasks, 22 evaluation aspects, and corresponding 37 datasets demonstrate that this approach can effectively allow us to achieve what one desires to evaluate for texts simply by natural language instructions. This nature helps us overcome several long-standing challenges in text evaluation--how to achieve customized, multi-faceted evaluation without the need for annotated samples. We make our code publicly available at https://github.com/jinlanfu/GPTScore.

Key facts. Cited by 357 papers in the map (data through 7 Oct 2026). Builds on 43 of its 70 references that are in the map. First posted on arXiv 8 Feb 2023.

Builds on

43 of 70 references.

References · 43

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  6. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  7. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  8. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  9. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  10. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  11. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
  12. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
31 more
  1. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  2. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,413
  3. 2020BLEURT: Learning Robust Metrics for Text Generationcited by 1,196
  4. 2015Teaching Machines to Read and Comprehendcited by 1,039
  5. 2020COMET: A Neural Framework for MT Evaluationcited by 1,024
  6. 2022Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Taskscited by 858
  7. 2021BARTScore: Evaluating Generated Text as Text Generationcited by 719
  8. 2020SummEval: Re-evaluating Summarization Evaluationcited by 666
  9. 2020Towards a Human-like Open-Domain Chatbotcited by 467
  10. 2019MoverScore: Text Generation Evaluating with Contextualized Embeddings and Earth Mover Distancecited by 415
  11. 2020Asking and Answering Questions to Evaluate the Factual Consistency of Summariescited by 376
  12. 2021Experts, Errors, and Context: A Large-Scale Study of Human Evaluation for Machine Translationcited by 333
  13. 2018Newsroom: A Dataset of 1.3 Million Summaries with Diverse Extractive Strategiescited by 292
  14. 2020FEQA: A Question Answering Evaluation Framework for Faithfulness Assessment in Abstractive Summarizationcited by 292
  15. 2022Towards a Unified Multi-Dimensional Evaluator for Text Generationcited by 264
  16. 2021QuestEval: Summarization Asks for Fact-based Evaluationcited by 241
  17. 2020USR: An Unsupervised and Reference Free Evaluation Metric for Dialog Generationcited by 186
  18. 2020Re-evaluating Evaluation in Text Summarizationcited by 144
  19. 2022Revisiting the Gold Standard: Grounding Summarization Evaluation with Robust Human Evaluationcited by 141
  20. 2020Unsupervised Evaluation of Interactive Dialog with DialoGPTcited by 139
  21. 2015Semantically Conditioned LSTM-based Natural Language Generation for Spoken Dialogue Systemscited by 134
  22. 2020Automatic Machine Translation Evaluation in Many Languages via Zero-Shot Paraphrasingcited by 114
  23. 2020Factual Error Correction for Abstractive Summarization Modelscited by 108
  24. 2021DynaEval: Unifying Turn and Dialogue Level Evaluationcited by 59
  25. 2021Conversations Are Not Flat: Modeling the Dynamic Information Flow across Dialogue Utterancescited by 40
  26. 2022DEAM: Dialogue Coherence Evaluation using AMR-based Semantic Manipulationscited by 25
  27. 2021Towards Quantifiable Dialogue Coherence Evaluationcited by 21
  28. 2022Not All Errors are Equal: Learning Text Generation Metrics using Stratified Error Synthesiscited by 18
  29. 2021Cut the CARP: Fishing for zero-shot story evaluationcited by 14
  30. 2022Robust Preference Learning for Storytelling via Contrastive Reinforcement Learningcited by 14
  31. 2022FineD-Eval: Fine-grained Automatic Dialogue-Level Evaluationcited by 13

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 112; 2024: 101; 2025: 91; 2026: 53.

357 papers.

Sort:

By area:

  1. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  2. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,317
  3. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,205
  4. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,221
  5. 2023Instruction-Following Evaluation for Large Language Modelscited by 1,143
  6. 2023SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Modelscited by 787
  7. 2023CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquingcited by 751
  8. 2023ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debatecited by 724
  9. 2024LLM Evaluators Recognize and Favor Their Own Generationscited by 651
  10. 2024LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methodscited by 451
  11. 2023Is ChatGPT a Good NLG Evaluator? A Preliminary Studycited by 420
  12. 2023Harnessing the Power of LLMs in Practice: A Survey on ChatGPT and Beyondcited by 409
38 more
  1. 2023Ragas: Automated Evaluation of Retrieval Augmented Generationcited by 408
  2. 2023Evaluating Large Language Models at Evaluating Instruction Followingcited by 290
  3. 2025PaperBench: Evaluating AI's Ability to Replicate AI Researchcited by 286
  4. 2023AlignScore: Evaluating Factual Consistency with a Unified Alignment Functioncited by 284
  5. 2023Challenges and Applications of Large Language Modelscited by 251
  6. 2023Large Language Model Alignment: A Surveycited by 241
  7. 2023Driving with LLMs: Fusing Object-Level Vector Modality for Explainable Autonomous Drivingcited by 231
  8. 2024When Can LLMs Actually Correct Their Own Mistakes? A Critical Survey of Self-Correction of LLMscited by 230
  9. 2024Agent-as-a-Judge: Evaluate Agents with Agentscited by 223
  10. 2023Survey on Factuality in Large Language Models: Knowledge, Retrieval and Domain-Specificitycited by 204
  11. 2023Evaluating Large Language Models: A Comprehensive Surveycited by 200
  12. 2024VoiceBench: Benchmarking LLM-Based Voice Assistantscited by 192
  13. 2023OctoPack: Instruction Tuning Code Large Language Modelscited by 178
  14. 2023RoleLLM: Benchmarking, Eliciting, and Enhancing Role-Playing Abilities of Large Language Modelscited by 173
  15. 2023Benchmarking Foundation Models with Language-Model-as-an-Examinercited by 172
  16. 2025Which Agent Causes Task Failures and When? On Automated Failure Attribution of LLM Multi-Agent Systemscited by 170
  17. 2023Large Language Model Routing with Benchmark Datasetscited by 166
  18. 2023Generative Judge for Evaluating Alignmentcited by 163
  19. 2023UniChart: A Universal Vision-language Pretrained Model for Chart Comprehension and Reasoningcited by 158
  20. 2024Aligning with Human Judgement: The Role of Pairwise Preference in Large Language Model Evaluatorscited by 146
  21. 2023BooookScore: A systematic exploration of book-length summarization in the era of LLMscited by 137
  22. 2024InfiAgent-DABench: Evaluating Agents on Data Analysis Taskscited by 134
  23. 2023Hallucinations in Large Multilingual Translation Modelscited by 126
  24. 2023Factuality Challenges in the Era of Large Language Modelscited by 125
  25. 2023Opportunities and Challenges for ChatGPT and Large Language Models in Biomedicine and Healthcited by 118
  26. 2023ChartBench: A Benchmark for Complex Visual Reasoning in Chartscited by 116
  27. 2023PRD: Peer Rank and Discussion Improve Large Language Model based Evaluationscited by 114
  28. 2024Pride and Prejudice: LLM Amplifies Self-Bias in Self-Refinementcited by 111
  29. 2024A Comprehensive Survey on Process-Oriented Automatic Text Summarization with Exploration of LLM-Based Methodscited by 101
  30. 2023LLM-Eval: Unified Multi-Dimensional Automatic Evaluation for Open-Domain Conversations with Large Language Modelscited by 100
  31. 2024DiscoveryBench: Towards Data-Driven Discovery with Large Language Modelscited by 97
  32. 2023Shepherd: A Critic for Language Model Generationcited by 96
  33. 2023Wider and Deeper LLM Networks are Fairer LLM Evaluatorscited by 91
  34. 2024From Matching to Generation: A Survey on Generative Information Retrievalcited by 89
  35. 2023LLMs as Narcissistic Evaluators: When Ego Inflates Evaluation Scorescited by 88
  36. 2023Are Large Language Model-based Evaluators the Solution to Scaling Up Multilingual Evaluation?cited by 87
  37. 2023Exploring the Use of Large Language Models for Reference-Free Text Quality Evaluation: An Empirical Studycited by 85
  38. 2023LLMScore: Unveiling the Power of Large Language Models in Text-to-Image Synthesis Evaluationcited by 85

50 most cited of 357 All 357 (JSON)