# BatchEval: Towards Human-like Text Evaluation

arXiv: https://arxiv.org/abs/2401.00437 · HTML: https://mlsanity.com/paper/2401.00437/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 34 references that are in the map.
- First posted on arXiv 31 Dec 2023.

## Builds on (most cited first, 19 of 19)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment](https://mlsanity.com/paper/2303.16634/) (2023)
- [Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarization](https://mlsanity.com/paper/1808.08745/) (2018)
- [Teaching Machines to Read and Comprehend](https://mlsanity.com/paper/1506.03340/) (2015)
- [AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedback](https://mlsanity.com/paper/2305.14387/) (2023)
- [Can Large Language Models Be an Alternative to Human Evaluations?](https://mlsanity.com/paper/2305.01937/) (2023)
- [Asking and Answering Questions to Evaluate the Factual Consistency of Summaries](https://mlsanity.com/paper/2004.04228/) (2020)
- [Evaluating Large Language Models: A Comprehensive Survey](https://mlsanity.com/paper/2310.19736/) (2023)
- [USR: An Unsupervised and Reference Free Evaluation Metric for Dialog Generation](https://mlsanity.com/paper/2005.00456/) (2020)
- [A Survey of Evaluation Metrics Used for NLG Systems](https://mlsanity.com/paper/2008.12009/) (2020)
- [Unsupervised Evaluation of Interactive Dialog with DialoGPT](https://mlsanity.com/paper/2006.12719/) (2020)
- [Annotating and Modeling Fine-grained Factuality in Summarization](https://mlsanity.com/paper/2104.04302/) (2021)
- [Towards an Automatic Turing Test: Learning to Evaluate Dialogue Responses](https://mlsanity.com/paper/1708.07149/) (2017)
- [Of Human Criteria and Automatic Metrics: A Benchmark of the Evaluation of Story Generation](https://mlsanity.com/paper/2208.11646/) (2022)
- [DynaEval: Unifying Turn and Dialogue Level Evaluation](https://mlsanity.com/paper/2106.01112/) (2021)
- [MME-CRS: Multi-Metric Evaluation Based on Correlation Re-Scaling for Evaluating Open-Domain Dialogue](https://mlsanity.com/paper/2206.09403/) (2022)

## Built on by (most cited first, 17 of 17)

- [From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judge](https://mlsanity.com/paper/2411.16594/) (2024)
- [LLM-based NLG Evaluation: Current Status and Challenges](https://mlsanity.com/paper/2402.01383/) (2024)
- [Can LLMs Replace Human Evaluators? An Empirical Study of LLM-as-a-Judge in Software Engineering](https://mlsanity.com/paper/2502.06193/) (2025)
- [Integrate the Essence and Eliminate the Dross: Fine-Grained Self-Consistency for Free-Form Language Generation](https://mlsanity.com/paper/2407.02056/) (2024)
- [Large Language Model as an Assignment Evaluator: Insights, Feedback, and Challenges in a 1000+ Student Course](https://mlsanity.com/paper/2407.05216/) (2024)
- [CPO: Addressing Reward Ambiguity in Role-playing Dialogue via Comparative Policy Optimization](https://mlsanity.com/paper/2508.09074/) (2025)
- [Focused Large Language Models are Stable Many-Shot Learners](https://mlsanity.com/paper/2408.13987/) (2024)
- [Can LLM-as-a-Judge Reliably Verify Rubrics in Agentic Scenarios?](https://mlsanity.com/paper/2606.29920/) (2026)
- [Large Language Models Are Active Critics in NLG Evaluation](https://mlsanity.com/paper/2410.10724/) (2024)
- [Human Evaluation of Procedural Knowledge Graph Extraction from Text with Large Language Models](https://mlsanity.com/paper/2412.03589/) (2024)
- [BatchGEMBA: Token-Efficient Machine Translation Evaluation with Batched Prompting and Prompt Compression](https://mlsanity.com/paper/2503.02756/) (2025)
- [EmoHarbor: Evaluating Personalized Emotional Support by Simulating the User's Internal World](https://mlsanity.com/paper/2601.01530/) (2026)
- [Poor-Supervised Evaluation for SuperLLM via Mutual Consistency](https://mlsanity.com/paper/2408.13738/) (2024)
- [Can Large Language Models Outperform Non-Experts in Poetry Evaluation? A Comparative Study Using the Consensual Assessment Technique](https://mlsanity.com/paper/2502.19064/) (2025)
- [LecEval: An Automated Metric for Multimodal Knowledge Acquisition in Multimedia Learning](https://mlsanity.com/paper/2505.02078/) (2025)
- [How to Engage Your Readers? Generating Guiding Questions to Promote Active Reading](https://mlsanity.com/paper/2407.14309/) (2024)
- [PsyEvo: A Personalized Counseling Agent That Self-Evolves at Test Time](https://mlsanity.com/paper/2610.02885/) (2026)

## Abstract (arXiv)

Significant progress has been made in automatic text evaluation with the introduction of large language models (LLMs) as evaluators. However, current sample-wise evaluation paradigm suffers from the following issues: (1) Sensitive to prompt design; (2) Poor resistance to noise; (3) Inferior ensemble performance with static reference. Inspired by the fact that humans treat both criterion definition and inter sample comparison as references for evaluation, we propose BatchEval, a paradigm that conducts batch-wise evaluation iteratively to alleviate the above problems. We explore variants under this paradigm and confirm the optimal settings are two stage procedure with heterogeneous batch composition strategy and decimal scoring format. Comprehensive experiments across 3 LLMs on 4 text evaluation tasks demonstrate that BatchEval outperforms state-of-the-art methods by 10.5% on Pearson correlations with only 64% API cost on average. Further analyses have been conducted to verify the robustness, generalization, and working mechanism of BatchEval.