mlsanityData to 14 Oct 2026

Evaluation Metrics in the Era of GPT-4: Reliably Evaluating Large Language Models on Sequence to Sequence Tasks

arXiv 2310.13800 ↗First posted 20 Oct 2023cs.CLcited by 53

Not on the map. Why?

Abstract (arXiv)

Large Language Models (LLMs) evaluation is a patchy and inconsistent landscape, and it is becoming clear that the quality of automatic evaluation metrics is not keeping up with the pace of development of generative models. We aim to improve the understanding of current models' performance by providing a preliminary and hybrid evaluation on a range of open and closed-source generative LLMs on three NLP benchmarks: text summarisation, text simplification and grammatical error correction (GEC), using both automatic and human evaluation. We also explore the potential of the recently released GPT-4 to act as an evaluator. We find that ChatGPT consistently outperforms many other popular models according to human reviewers on the majority of metrics, while scoring much more poorly when using classic automatic evaluation metrics. We also find that human reviewers rate the gold reference as much worse than the best models' outputs, indicating the poor quality of many popular benchmarks. Finally, we find that GPT-4 is capable of ranking models' outputs in a way which aligns reasonably closely to human judgement despite task-specific variations, with a lower alignment in the GEC task.

Key facts. Cited by 53 papers in the map (data through 14 Oct 2026). Builds on 20 of its 44 references that are in the map. First posted on arXiv 20 Oct 2023.

Builds on

20 of 44 references.

References · 20

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2022Training language models to follow instructions with human feedbackcited by 15,971
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  6. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,470
  7. 2023Large Language Models are not Fair Evaluatorscited by 1,085
  8. 2015Teaching Machines to Read and Comprehendcited by 1,049
  9. 2023A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivitycited by 894
  10. 2023ChatGPT Outperforms Crowd-Workers for Text-Annotation Taskscited by 870
  11. 2016Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyondcited by 812
  12. 2023Can Large Language Models Be an Alternative to Human Evaluations?cited by 734
8 more
  1. 2020SummEval: Re-evaluating Summarization Evaluationcited by 674
  2. 2022UL2: Unifying Language Learning Paradigmscited by 295
  3. 2022OPT-IML: Scaling Language Model Instruction Meta Learning through the Lens of Generalizationcited by 236
  4. 2020Neural CRF Model for Sentence Alignment in Text Simplificationcited by 106
  5. 2023Is ChatGPT a Highly Fluent Grammatical Error Correction System? A Comprehensive Evaluationcited by 97
  6. 2023ChatGPT or Grammarly? Evaluating ChatGPT on Grammatical Error Correction Benchmarkcited by 77
  7. 2022POTATO: The Portable Text Annotation Toolcited by 64
  8. 2022Investigating Efficiently Extending Transformers for Long Input Summarizationcited by 43

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 2; 2024: 29; 2025: 16; 2026: 6.

53 papers.

Sort:

By area:

  1. 2024LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methodscited by 465
  2. 2024Self-Preference Bias in LLM-as-a-Judgecited by 258
  3. 2024Judging the Judges: Evaluating Alignment and Vulnerabilities in LLMs-as-Judgescited by 205
  4. 2024Superfiltering: Weak-to-Strong Data Filtering for Fast Instruction-Tuningcited by 138
  5. 2024Trust or Escalate: LLM Judges with Provable Guarantees for Human Agreementcited by 105
  6. 2024FinTextQA: A Dataset for Long-form Financial Question Answeringcited by 54
  7. 2023Extending Context Window of Large Language Models via Semantic Compressioncited by 45
  8. 2025Evaluating Scoring Bias in LLM-as-a-Judgecited by 43
  9. 2024Revisiting Knowledge Distillation for Autoregressive Language Modelscited by 42
  10. 2024Leveraging Large Language Models for NLG Evaluation: Advances and Challengescited by 37
  11. 2024Are LLM-based Evaluators Confusing NLG Quality Criteria?cited by 36
  12. 2024CopyBench: Measuring Literal and Non-Literal Reproduction of Copyright-Protected Text in Language Model Generationcited by 35
38 more
  1. 2025Towards Reasoning Ability of Small Language Modelscited by 29
  2. 2024Towards Completeness-Oriented Tool Retrieval for Large Language Modelscited by 26
  3. 2024Explainable Generative AI (GenXAI): A Survey, Conceptualization, and Research Agendacited by 22
  4. 2025ReLearn: Unlearning via Learning for Large Language Modelscited by 22
  5. 2025TrustJudge: Inconsistencies of LLM-as-a-Judge and How to Alleviate Themcited by 20
  6. 2024Themis: A Reference-free NLG Evaluation Language Model with Flexibility and Interpretabilitycited by 19
  7. 2024Healthcare Copilot: Eliciting the Power of General LLMs for Medical Consultationcited by 17
  8. 2024Beyond Traditional Benchmarks: Analyzing Behaviors of Open LLMs on Data-to-Text Generationcited by 16
  9. 2024A User-Centric Multi-Intent Benchmark for Evaluating Large Language Modelscited by 15
  10. 2024CRAFT Your Dataset: Task-Specific Synthetic Dataset Generation Through Corpus Retrieval and Augmentationcited by 14
  11. 2025Bone Soups: A Seek-and-Soup Model Merging Approach for Controllable Multi-Objective Generationcited by 14
  12. 2024Uncertainty Aware Learning for Language Model Alignmentcited by 13
  13. 2024ARES: Alternating Reinforcement Learning and Supervised Fine-Tuning for Enhanced Multi-Modal Chain-of-Thought Reasoning Through Diverse AI Feedbackcited by 13
  14. 2024RuleR: Improving LLM Controllability by Rule-based Data Recyclingcited by 11
  15. 2024CharacterGPT: A Persona Reconstruction Framework for Role-Playing Agentscited by 10
  16. 2024mCSQA: Multilingual Commonsense Reasoning Dataset with Unified Creation Strategy by Language Models and Humanscited by 10
  17. 2024Rethinking the Roles of Large Language Models in Chinese Grammatical Error Correctioncited by 9
  18. 2024Mosaic-IT: Cost-Free Compositional Data Synthesis for Instruction Tuningcited by 9
  19. 2024SimulBench: Evaluating Language Models with Creative Simulation Taskscited by 8
  20. 2025Judge as A Judge: Improving the Evaluation of Retrieval-Augmented Generation through the Judge-Consistency of Large Language Modelscited by 8
  21. 2025Rubrik's Cube: Testing a New Rubric for Evaluating Explanations on the CUBE datasetcited by 8
  22. 2024Copyright-Protected Language Generation via Adaptive Model Fusioncited by 7
  23. 2025LLMs as Span Annotators: A Comparative Study of LLMs and Humanscited by 7
  24. 2025Discerning minds or generic tutors? Evaluating instructional guidance capabilities in Socratic LLMscited by 5
  25. 2023LLM-SAP: Large Language Models Situational Awareness Based Planningcited by 4
  26. 2024Large Language Models Are State-of-the-Art Evaluator for Grammatical Error Correctioncited by 3
  27. 2025LLM-as-a-Judge: Rapid Evaluation of Legal Document Recommendation for Retrieval-Augmented Generationcited by 3
  28. 2024WIKIGENBENCH: Exploring Full-length Wikipedia Generation under Real-World Scenariocited by 2
  29. 2024DSGram: Dynamic Weighting Sub-Metrics for Grammatical Error Correction in the Era of Large Language Modelscited by 1
  30. 2025Adaptive Repetition for Mitigating Position Bias in LLM-Based Rankingcited by 1
  31. 2025The illusion of a perfect metric: Why evaluating AI's words is harder than it lookscited by 1
  32. 2025AI-in-the-Loop: Privacy Preserving Real-Time Scam Detection and Conversational Scambaiting by Leveraging LLMs and Federated Learningcited by 1
  33. 2026Stability vs. Manipulability: Evaluating Robustness Under Post-Decision Interaction in LLM Judgescited by 1
  34. 2026Ideation Arena: Evaluating LLM Generated Research Ideas with Battle-style Human Expert Assessmentcited by 1
  35. 2026When Consistency Does Not Mean Reliability: Evaluating Local LLM Judges Against Human Ratingscited by 1
  36. 2025Enhancing Faithfulness in Abstractive Summarization via Span-Level Fine-Tuningcited by 0
  37. 2025Leveraging Hierarchical Organization for Medical Multi-document Summarizationcited by 0
  38. 2025Subjective Question Generation and Answer Evaluation using NLPcited by 0

50 most cited of 53 All 53 (JSON)