mlsanityData to 7 Oct 2026

BatchEval: Towards Human-like Text Evaluation

arXiv 2401.00437 ↗First posted 31 Dec 2023cs.CLcited by 16

Not on the map. Why?

Abstract (arXiv)

Significant progress has been made in automatic text evaluation with the introduction of large language models (LLMs) as evaluators. However, current sample-wise evaluation paradigm suffers from the following issues: (1) Sensitive to prompt design; (2) Poor resistance to noise; (3) Inferior ensemble performance with static reference. Inspired by the fact that humans treat both criterion definition and inter sample comparison as references for evaluation, we propose BatchEval, a paradigm that conducts batch-wise evaluation iteratively to alleviate the above problems. We explore variants under this paradigm and confirm the optimal settings are two stage procedure with heterogeneous batch composition strategy and decimal scoring format. Comprehensive experiments across 3 LLMs on 4 text evaluation tasks demonstrate that BatchEval outperforms state-of-the-art methods by 10.5% on Pearson correlations with only 64% API cost on average. Further analyses have been conducted to verify the robustness, generalization, and working mechanism of BatchEval.

Key facts. Cited by 16 papers in the map (data through 7 Oct 2026). Builds on 19 of its 34 references that are in the map. First posted on arXiv 31 Dec 2023.

Builds on

19 of 34 references.

References · 19

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  4. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  5. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,205
  6. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,308
  7. 2015Teaching Machines to Read and Comprehendcited by 1,039
  8. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 738
  9. 2023Can Large Language Models Be an Alternative to Human Evaluations?cited by 729
  10. 2020Asking and Answering Questions to Evaluate the Factual Consistency of Summariescited by 376
  11. 2023Evaluating Large Language Models: A Comprehensive Surveycited by 200
  12. 2020USR: An Unsupervised and Reference Free Evaluation Metric for Dialog Generationcited by 186
7 more
  1. 2020A Survey of Evaluation Metrics Used for NLG Systemscited by 164
  2. 2020Unsupervised Evaluation of Interactive Dialog with DialoGPTcited by 139
  3. 2021Annotating and Modeling Fine-grained Factuality in Summarizationcited by 131
  4. 2017Towards an Automatic Turing Test: Learning to Evaluate Dialogue Responsescited by 98
  5. 2022Of Human Criteria and Automatic Metrics: A Benchmark of the Evaluation of Story Generationcited by 73
  6. 2021DynaEval: Unifying Turn and Dialogue Level Evaluationcited by 59
  7. 2022MME-CRS: Multi-Metric Evaluation Based on Correlation Re-Scaling for Evaluating Open-Domain Dialoguecited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 9; 2025: 5; 2026: 2.

16 papers.

Sort:

By area:

  1. 2024From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judgecited by 480
  2. 2024LLM-based NLG Evaluation: Current Status and Challengescited by 71
  3. 2025Can LLMs Replace Human Evaluators? An Empirical Study of LLM-as-a-Judge in Software Engineeringcited by 64
  4. 2024Integrate the Essence and Eliminate the Dross: Fine-Grained Self-Consistency for Free-Form Language Generationcited by 21
  5. 2024Large Language Model as an Assignment Evaluator: Insights, Feedback, and Challenges in a 1000+ Student Coursecited by 19
  6. 2025CPO: Addressing Reward Ambiguity in Role-playing Dialogue via Comparative Policy Optimizationcited by 10
  7. 2024Focused Large Language Models are Stable Many-Shot Learnerscited by 8
  8. 2024Large Language Models Are Active Critics in NLG Evaluationcited by 6
  9. 2026Can LLM-as-a-Judge Reliably Verify Rubrics in Agentic Scenarios?cited by 5
  10. 2024Human Evaluation of Procedural Knowledge Graph Extraction from Text with Large Language Modelscited by 4
  11. 2025BatchGEMBA: Token-Efficient Machine Translation Evaluation with Batched Prompting and Prompt Compressioncited by 3
  12. 2026EmoHarbor: Evaluating Personalized Emotional Support by Simulating the User's Internal Worldcited by 3
4 more
  1. 2024Poor-Supervised Evaluation for SuperLLM via Mutual Consistencycited by 1
  2. 2025Can Large Language Models Outperform Non-Experts in Poetry Evaluation? A Comparative Study Using the Consensual Assessment Techniquecited by 1
  3. 2025LecEval: An Automated Metric for Multimodal Knowledge Acquisition in Multimedia Learningcited by 1
  4. 2024How to Engage Your Readers? Generating Guiding Questions to Promote Active Readingcited by 0