mlsanityData to 7 Oct 2026

Can Large Language Models Be an Alternative to Human Evaluations?

arXiv 2305.01937 ↗First posted 3 May 2023cs.CL cs.HCcited by 729

Not on the map. Why?

Cited by 60 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Human evaluation is indispensable and inevitable for assessing the quality of texts generated by machine learning models or written by humans. However, human evaluation is very difficult to reproduce and its quality is notoriously unstable, hindering fair comparisons among different natural language processing (NLP) models and algorithms. Recently, large language models (LLMs) have demonstrated exceptional performance on unseen tasks when only the task instructions are provided. In this paper, we explore if such an ability of the LLMs can be used as an alternative to human evaluation. We present the LLMs with the exact same instructions, samples to be evaluated, and questions used to conduct human evaluation, and then ask the LLMs to generate responses to those questions; we dub this LLM evaluation. We use human evaluation and LLM evaluation to evaluate the texts in two NLP tasks: open-ended story generation and adversarial attacks. We show that the result of LLM evaluation is consistent with the results obtained by expert human evaluation: the texts rated higher by human experts are also rated higher by the LLMs. We also find that the results of LLM evaluation are stable over different formatting of the task instructions and the sampling algorithm used to generate the answer. We are the first to show the potential of using LLMs to assess the quality of texts and discuss the limitations and ethical considerations of LLM evaluation.

Key facts. Cited by 729 papers in the map (data through 7 Oct 2026). Builds on 31 of its 56 references that are in the map. First posted on arXiv 3 May 2023.

Builds on

31 of 56 references.

References · 31

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  6. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  7. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  8. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  9. 2015Character-level Convolutional Networks for Text Classificationcited by 2,293
  10. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  11. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  12. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,371
19 more
  1. 2018Hierarchical Neural Story Generationcited by 1,182
  2. 2022Red Teaming Language Models with Language Modelscited by 983
  3. 2023ChatGPT Outperforms Crowd-Workers for Text-Annotation Taskscited by 849
  4. 2020Aligning AI With Shared Human Valuescited by 706
  5. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learnedcited by 702
  6. 2019Is BERT Really Robust? A Strong Baseline for Natural Language Attack on Text Classification and Entailmentcited by 663
  7. 2021Editing Factual Knowledge in Language Modelscited by 548
  8. 2022Socratic Models: Composing Zero-Shot Multimodal Reasoning with Languagecited by 513
  9. 2018Generating Natural Language Adversarial Examplescited by 422
  10. 2023Is ChatGPT a Good NLG Evaluator? A Preliminary Studycited by 420
  11. 2020BAE: BERT-based Adversarial Examples for Text Classificationcited by 316
  12. 2021All That's 'Human' Is Not Gold: Evaluating Human Evaluation of Generated Textcited by 287
  13. 2023Is ChatGPT better than Human Annotators? Potential and Limitations of ChatGPT in Explaining Implicit Hate Speechcited by 165
  14. 2020A Knowledge-Enhanced Pretraining Model for Commonsense Story Generationcited by 116
  15. 2021The Perils of Using Mechanical Turk to Evaluate Open-Ended Text Generationcited by 96
  16. 2020Reevaluating Adversarial Examples in Natural Languagecited by 83
  17. 2022Detection of Word Adversarial Examples in Text Classification: Benchmark and Baseline via Robust Density Estimationcited by 32
  18. 2019Improving Neural Story Generation by Targeted Common Sense Groundingcited by 16
  19. 2021BERT is Robust! A Case Against Synonym-Based Adversarial Examples in Text Classificationcited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 95; 2024: 287; 2025: 214; 2026: 133.

729 papers.

Sort:

By area:

  1. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  2. 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,582
  3. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,221
  4. 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,151
  5. 2023Instruction-Following Evaluation for Large Language Modelscited by 1,143
  6. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 738
  7. 2023ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debatecited by 724
  8. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedbackcited by 630
  9. 2023Prometheus: Inducing Fine-grained Evaluation Capability in Language Modelscited by 558
  10. 2023The Falcon Series of Open Language Modelscited by 526
  11. 2024From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judgecited by 480
  12. 2023GPT-4 Is Too Smart To Be Safe: Stealthy Chat with LLMs via Ciphercited by 401
38 more
  1. 2023Large Language Models for Software Engineering: A Systematic Literature Reviewcited by 389
  2. 2023DoLa: Decoding by Contrasting Layers Improves Factuality in Large Language Modelscited by 367
  3. 2024MLLM-as-a-Judge: Assessing Multimodal LLM-as-a-Judge with Vision-Language Benchmarkcited by 362
  4. 2023Evaluating Large Language Models at Evaluating Instruction Followingcited by 290
  5. 2025PaperBench: Evaluating AI's Ability to Replicate AI Researchcited by 286
  6. 2024Judging the Judges: A Systematic Study of Position Bias in LLM-as-a-Judgecited by 249
  7. 2024LogicVista: Multimodal LLM Logical Reasoning Benchmark in Visual Contextscited by 240
  8. 2024LLMs instead of Human Judges? A Large Scale Empirical Study across 20 NLP Evaluation Taskscited by 236
  9. 2024When Can LLMs Actually Correct Their Own Mistakes? A Critical Survey of Self-Correction of LLMscited by 230
  10. 2023PersonaLLM: Investigating the Ability of Large Language Models to Express Personality Traitscited by 229
  11. 2024Leak, Cheat, Repeat: Data Contamination and Evaluation Malpractices in Closed-Source LLMscited by 224
  12. 2024Two Tales of Persona in LLMs: A Survey of Role-Playing and Personalizationcited by 203
  13. 2023FLASK: Fine-grained Language Model Evaluation based on Alignment Skill Setscited by 198
  14. 2024Judging the Judges: Evaluating Alignment and Vulnerabilities in LLMs-as-Judgescited by 197
  15. 2023Honeybee: Locality-enhanced Projector for Multimodal LLMcited by 195
  16. 2023Knowledge Editing for Large Language Models: A Surveycited by 189
  17. 2023Evaluating Correctness and Faithfulness of Instruction-Following Models for Question Answeringcited by 150
  18. 2024Aligning with Human Judgement: The Role of Pairwise Preference in Large Language Model Evaluatorscited by 146
  19. 2024A Human-Inspired Reading Agent with Gist Memory of Very Long Contextscited by 140
  20. 2024Lookback Lens: Detecting and Mitigating Contextual Hallucinations in Large Language Models Using Only Attention Mapscited by 140
  21. 2023Large language models can accurately predict searcher preferencescited by 129
  22. 2023SmartPlay: A Benchmark for LLMs as Intelligent Agentscited by 123
  23. 2024Optimization-based Prompt Injection Attack to LLM-as-a-Judgecited by 123
  24. 2024RAGBench: Explainable Benchmark for Retrieval-Augmented Generation Systemscited by 115
  25. 2023Branch-Solve-Merge Improves Large Language Model Evaluation and Generationcited by 104
  26. 2024Large Language Models Must Be Taught to Know What They Don't Knowcited by 99
  27. 2024Trust or Escalate: LLM Judges with Provable Guarantees for Human Agreementcited by 99
  28. 2024Bias and Unfairness in Information Retrieval Systems: New Challenges in the LLM Eracited by 97
  29. 2023Jatmo: Prompt Injection Defense by Task-Specific Finetuningcited by 89
  30. 2023Are Large Language Model-based Evaluators the Solution to Scaling Up Multilingual Evaluation?cited by 87
  31. 2024A Systematic Survey and Critical Review on Evaluating Large Language Models: Challenges, Limitations, and Recommendationscited by 84
  32. 2024LLM-Rubric: A Multidimensional, Calibrated Approach to Automated Evaluation of Natural Language Textscited by 81
  33. 2023On Learning to Summarize with Large Language Models as Referencescited by 80
  34. 2024RAG vs Fine-tuning: Pipelines, Tradeoffs, and a Case Study on Agriculturecited by 80
  35. 2024Watermark Stealing in Large Language Modelscited by 79
  36. 2023EvalLM: Interactive Evaluation of Large Language Model Prompts on User-Defined Criteriacited by 73
  37. 2024Foundational Autoraters: Taming Large Language Models for Better Automatic Evaluationcited by 72
  38. 2025The Alternative Annotator Test for LLM-as-a-Judge: How to Statistically Justify Replacing Human Annotators with LLMscited by 72

50 most cited of 729 All 729 (JSON)