mlsanityData to 7 Oct 2026

A Closer Look into Automatic Evaluation Using Large Language Models

arXiv 2310.05657 ↗First posted 9 Oct 2023cs.CLcited by 17

Not on the map. Why?

Abstract (arXiv)

Using large language models (LLMs) to evaluate text quality has recently gained popularity. Some prior works explore the idea of using LLMs for evaluation, while they differ in some details of the evaluation process. In this paper, we analyze LLM evaluation (Chiang and Lee, 2023) and G-Eval (Liu et al., 2023), and we discuss how those details in the evaluation process change how well the ratings given by LLMs correlate with human ratings. We find that the auto Chain-of-Thought (CoT) used in G-Eval does not always make G-Eval more aligned with human ratings. We also show that forcing the LLM to output only a numeric rating, as in G-Eval, is suboptimal. Last, we reveal that asking the LLM to explain its own ratings consistently improves the correlation between the ChatGPT and human ratings and pushes state-of-the-art (SoTA) correlations on two meta-evaluation datasets.

Key facts. Cited by 17 papers in the map (data through 7 Oct 2026). Builds on 17 of its 27 references that are in the map. First posted on arXiv 9 Oct 2023.

Builds on

17 of 27 references.

References · 17

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  3. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  4. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  5. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  6. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,205
  7. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  8. 2015Teaching Machines to Read and Comprehendcited by 1,039
  9. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,023
  10. 2023Can Large Language Models Be an Alternative to Human Evaluations?cited by 729
  11. 2020SummEval: Re-evaluating Summarization Evaluationcited by 666
  12. 2022Socratic Models: Composing Zero-Shot Multimodal Reasoning with Languagecited by 513
5 more
  1. 2023Is ChatGPT a Good NLG Evaluator? A Preliminary Studycited by 420
  2. 2022Towards a Unified Multi-Dimensional Evaluator for Text Generationcited by 264
  3. 2020USR: An Unsupervised and Reference Free Evaluation Metric for Dialog Generationcited by 186
  4. 2023Is ChatGPT better than Human Annotators? Potential and Limitations of ChatGPT in Explaining Implicit Hate Speechcited by 165
  5. 2023Topical-Chat: Towards Knowledge-Grounded Open-Domain Conversationscited by 88

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 10; 2025: 2; 2026: 4.

17 papers.

Sort:

By area:

  1. 2024LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methodscited by 451
  2. 2024MLLM-as-a-Judge: Assessing Multimodal LLM-as-a-Judge with Vision-Language Benchmarkcited by 362
  3. 2024MJ-Bench: Is Your Multimodal Reward Model Really a Good Judge for Text-to-Image Generation?cited by 64
  4. 2024Speech-Copilot: Leveraging Large Language Models for Speech Processing via Task Decomposition, Modularization, and Program Generationcited by 27
  5. 2023DEsignBench: Exploring and Benchmarking DALL-E 3 for Imagining Visual Designcited by 17
  6. 2025BadJudge: Backdoor Vulnerabilities of LLM-as-a-Judgecited by 13
  7. 2024Large Language Models for Social Networks: Applications, Challenges, and Solutionscited by 11
  8. 2024REAL Sampling: Boosting Factuality and Diversity of Open-Ended Generation via Asymptotic Entropycited by 11
  9. 2024Report Cards: Qualitative Evaluation of Language Models Using Natural Language Summariescited by 6
  10. 2024Aggregation of Reasoning: A Hierarchical Framework for Enhancing Answer Selection in Large Language Modelscited by 5
  11. 2024Do LLMs Agree on the Creativity Evaluation of Alternative Uses?cited by 5
  12. 2026Judge, Then Drive: A Critic-Centric Vision Language Action Framework for Autonomous Drivingcited by 3
5 more
  1. 2024ReFeR: Improving Evaluation and Reasoning through Hierarchy of Modelscited by 0
  2. 2025Enhancing Instruction-Following Capabilities in Seq2Seq Models: DoLA Adaptations for T5cited by 0
  3. 2026PEEM: Prompt Engineering Evaluation Metrics for Interpretable Joint Evaluation of Prompts and Responsescited by 0
  4. 2026Towards Annotation-Free Validation of MLLMs: A Vision-Language Logical Consistency Metriccited by 0
  5. 2026Operationalising Multi-Dimensional Evaluation for Conversational Agents: A Scalable, Governed Pipeline with Selective Re-evaluation and Model Benchmarkingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Aligning with Human Judgement: The Role of Pairwise Preference in Large Language Model Evaluators
  2. 2023Exploring the Reliability of Large Language Models as Customized Evaluators for Diverse NLP Tasks
  3. 2024HD-Eval: Aligning Large Language Model Evaluators Through Hierarchical Criteria Decomposition
  4. 2024METAL: Towards Multilingual Meta-Evaluation
  5. 2024MM-Eval: A Multilingual Meta-Evaluation Benchmark for LLM-as-a-Judge and Reward Models
  6. 2024Can Large Language Models be Trusted for Evaluation? Scalable Meta-Evaluation of LLMs as Evaluators via Agent Debate
  7. 2024Enhancing LLM Evaluations: The Garbling Trick
  8. 2025Accelerating Unbiased LLM Evaluation via Synthetic Feedback