mlsanityData to 7 Oct 2026

Exploring the Use of Large Language Models for Reference-Free Text Quality Evaluation: An Empirical Study

arXiv 2304.00723 ↗First posted 3 Apr 2023cs.CLcited by 85

Not on the map. Why?

Abstract (arXiv)

Evaluating the quality of generated text is a challenging task in NLP, due to the inherent complexity and diversity of text. Recently, large language models (LLMs) have garnered significant attention due to their impressive performance in various tasks. Therefore, we present this paper to investigate the effectiveness of LLMs, especially ChatGPT, and explore ways to optimize their use in assessing text quality. We compared three kinds of reference-free evaluation methods. The experimental results prove that ChatGPT is capable of evaluating text quality effectively from various perspectives without reference and demonstrates superior performance than most existing automatic metrics. In particular, the Explicit Score, which utilizes ChatGPT to generate a numeric score measuring text quality, is the most effective and reliable method among the three exploited approaches. However, directly comparing the quality of two texts may lead to suboptimal results. We believe this paper will provide valuable insights for evaluating text quality with LLMs and have released the used data.

Key facts. Cited by 85 papers in the map (data through 7 Oct 2026). Builds on 26 of its 40 references that are in the map. First posted on arXiv 3 Apr 2023.

Builds on

26 of 40 references.

References · 26

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  5. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  6. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  7. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  8. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,168
  9. 2021GLM: General Language Model Pretraining with Autoregressive Blank Infillingcited by 1,026
  10. 2021BARTScore: Evaluating Generated Text as Text Generationcited by 719
  11. 2020SummEval: Re-evaluating Summarization Evaluationcited by 666
  12. 2023Is ChatGPT a Good NLG Evaluator? A Preliminary Studycited by 420
14 more
  1. 2023Large Language Models Are State-of-the-Art Evaluators of Translation Qualitycited by 416
  2. 2019MoverScore: Text Generation Evaluating with Contextualized Embeddings and Earth Mover Distancecited by 415
  3. 2023GPTScore: Evaluate as You Desirecited by 357
  4. 2020USR: An Unsupervised and Reference Free Evaluation Metric for Dialog Generationcited by 186
  5. 2020Unsupervised Evaluation of Interactive Dialog with DialoGPTcited by 139
  6. 2020Automatic Machine Translation Evaluation in Many Languages via Zero-Shot Paraphrasingcited by 114
  7. 2021OpenMEVA: A Benchmark for Evaluating Open-ended Story Generation Metricscited by 61
  8. 2021DynaEval: Unifying Turn and Dialogue Level Evaluationcited by 59
  9. 2019Better Automatic Evaluation of Open-Domain Dialogue Systems with Contextualized Embeddingscited by 52
  10. 2022On the Evaluation Metrics for Paraphrase Generationcited by 32
  11. 2023Exploring ChatGPT's Ability to Rank Content: A Preliminary Study on Consistency with Human Preferencescited by 25
  12. 2023Is ChatGPT A Good Keyphrase Generator? A Preliminary Studycited by 16
  13. 2020Unsupervised Paraphrasing with Pretrained Language Modelscited by 15
  14. 2022SelF-Eval: Self-supervised Fine-grained Dialogue Evaluationcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 11; 2024: 33; 2025: 37; 2026: 4.

85 papers.

Sort:

By area:

  1. 2024From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judgecited by 480
  2. 2024LLMs instead of Human Judges? A Large Scale Empirical Study across 20 NLP Evaluation Taskscited by 236
  3. 2023Evaluating Large Language Models: A Comprehensive Surveycited by 200
  4. 2024Aligning with Human Judgement: The Role of Pairwise Preference in Large Language Model Evaluatorscited by 146
  5. 2024Is LLM-as-a-Judge Robust? Investigating Universal Adversarial Attacks on Zero-shot LLM Assessmentcited by 117
  6. 2023Jatmo: Prompt Injection Defense by Task-Specific Finetuningcited by 89
  7. 2023On Learning to Summarize with Large Language Models as Referencescited by 80
  8. 2024Systematic Evaluation of LLM-as-a-Judge in LLM Alignment Tasks: Explainable Metrics and Diverse Prompt Templatescited by 71
  9. 2023Corex: Pushing the Boundaries of Complex Reasoning through Multi-Model Collaborationcited by 70
  10. 2023Mark My Words: Analyzing and Evaluating Language Model Watermarkscited by 62
  11. 2023An Emulator for Fine-Tuning Large Language Models using Small Language Modelscited by 61
  12. 2024Evaluating LLMs at Detecting Errors in LLM Responsescited by 49
38 more
  1. 2024Evaluating the Evaluator: Measuring LLMs' Adherence to Task Evaluation Instructionscited by 34
  2. 2023Interactive Natural Language Processingcited by 31
  3. 2023Aligning Large Language Models with Human: A Surveycited by 31
  4. 2024Data-Centric Foundation Models in Computational Healthcare: A Surveycited by 27
  5. 2024Fairer Preferences Elicit Improved Human-Aligned Large Language Model Judgmentscited by 25
  6. 2024TRUCE: Private Benchmarking to Prevent Contamination and Improve Comparative Evaluation of LLMscited by 24
  7. 2025The Scales of Justitia: A Comprehensive Survey on Safety Evaluation of LLMscited by 24
  8. 2024Deconstructing The Ethics of Large Language Models from Long-standing Issues to New-emerging Dilemmas: A Surveycited by 23
  9. 2024RevisEval: Improving LLM-as-a-Judge via Response-Adapted Referencescited by 19
  10. 2025The Energy Loss Phenomenon in RLHF: A New Perspective on Mitigating Reward Hackingcited by 18
  11. 2024Language Models as Science Tutorscited by 17
  12. 2024What Makes a Good Story and How Can We Measure It? A Comprehensive Survey of Story Evaluationcited by 17
  13. 2025CoheMark: A Novel Sentence-Level Watermark for Enhanced Text Qualitycited by 17
  14. 2024One Prompt To Rule Them All: LLMs for Opinion Summary Evaluationcited by 16
  15. 2023Towards Explainable Evaluation Metrics for Machine Translationcited by 15
  16. 2024METAL: Towards Multilingual Meta-Evaluationcited by 14
  17. 2024What are the Essential Factors in Crafting Effective Long Context Multi-Hop Instruction Datasets? Insights and Best Practicescited by 14
  18. 2024Copilot Evaluation Harness: Evaluating LLM-Guided Software Programmingcited by 13
  19. 2025BadJudge: Backdoor Vulnerabilities of LLM-as-a-Judgecited by 13
  20. 2025Robust Reinforcement Learning from Human Feedback for Large Language Models Fine-Tuningcited by 13
  21. 2024Think Together and Work Better: Combining Humans' and LLMs' Think-Aloud Outcomes for Effective Text Evaluationcited by 12
  22. 2023Using Natural Language Explanations to Rescale Human Judgmentscited by 11
  23. 2025IntentionESC: An Intention-Centered Framework for Enhancing Emotional Support in Dialogue Systemscited by 11
  24. 2024The Fault in our Stars: Quality Assessment of Code Generation Benchmarkscited by 9
  25. 2025Decision Information Meets Large Language Models: The Future of Explainable Operations Researchcited by 9
  26. 2025Improving Preference Extraction In LLMs By Identifying Latent Knowledge Through Classifying Probescited by 9
  27. 2025From Generic Empathy to Personalized Emotional Support: A Self-Evolution Framework for User Preference Alignmentcited by 9
  28. 2024Rationale-Aware Answer Verification by Pairwise Self-Evaluationcited by 8
  29. 2025Bridging the LLM Accessibility Divide? Performance, Fairness, and Cost of Closed versus Open LLMs for Automated Essay Scoringcited by 7
  30. 2024Explingo: Explaining AI Predictions using Large Language Modelscited by 6
  31. 2025LLM-Enhanced Multiple Instance Learning for Joint Rumor and Stance Detection with Social Context Informationcited by 6
  32. 2025Rank-Then-Score: Enhancing Large Language Models for Automated Essay Scoringcited by 6
  33. 2024Aggregation of Reasoning: A Hierarchical Framework for Enhancing Answer Selection in Large Language Modelscited by 5
  34. 2024Evaluating LLMs for Targeted Concept Simplification for Domain-Specific Textscited by 5
  35. 2025HPSS: Heuristic Prompting Strategy Search for LLM Evaluatorscited by 5
  36. 2025A Multi-Task Evaluation of LLMs' Processing of Academic Text Inputcited by 5
  37. 2025Human Behavior Atlas: Benchmarking Unified Psychological and Social Behavior Understandingcited by 5
  38. 2025What Matters in Evaluating Book-Length Stories? A Systematic Study of Long Story Evaluationcited by 5

50 most cited of 85 All 85 (JSON)