mlsanityData to 7 Oct 2026

CheckEval: A reliable LLM-as-a-Judge framework for evaluating text generation using checklists

arXiv 2403.18771 ↗First posted 27 Mar 2024cs.CLcited by 87

Not on the map. Why?

Cited by 21 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Existing LLM-as-a-Judge approaches for evaluating text generation suffer from rating inconsistencies, with low agreement and high rating variance across different evaluator models. We attribute this to subjective evaluation criteria combined with Likert scale scoring in existing protocols. To address this issue, we introduce CheckEval, a checklist-based evaluation framework that improves rating reliability via decomposed binary questions. Through experiments with 12 evaluator models across multiple datasets, we first demonstrate that CheckEval strongly correlates with human judgments. More importantly, CheckEval dramatically improves the average agreement across evaluator models by 0.45 and reduces the score variance. CheckEval scores furthermore have the benefit of being more interpretable because it decomposes evaluation criteria into traceable binary decisions, allowing analyses of specific attributes driving quality judgments.

Key facts. Cited by 87 papers in the map (data through 7 Oct 2026). Builds on 32 of its 65 references that are in the map. First posted on arXiv 27 Mar 2024.

Builds on

32 of 65 references.

References · 32

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2022Training language models to follow instructions with human feedbackcited by 15,712
  5. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  6. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  7. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  8. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  9. 2023FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generationcited by 1,259
  10. 2020Beyond Accuracy: Behavioral Testing of NLP models with CheckListcited by 979
  11. 2023ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debatecited by 724
  12. 2021BARTScore: Evaluating Generated Text as Text Generationcited by 719
20 more
  1. 2020SummEval: Re-evaluating Summarization Evaluationcited by 666
  2. 2023Prometheus: Inducing Fine-grained Evaluation Capability in Language Modelscited by 558
  3. 2019MoverScore: Text Generation Evaluating with Contextualized Embeddings and Earth Mover Distancecited by 415
  4. 2020Asking and Answering Questions to Evaluate the Factual Consistency of Summariescited by 376
  5. 2023JudgeLM: Fine-tuned Large Language Models are Scalable Judgescited by 351
  6. 2023Large Language Models are Effective Text Rankers with Pairwise Ranking Promptingcited by 302
  7. 2022Towards a Unified Multi-Dimensional Evaluator for Text Generationcited by 264
  8. 2017Why We Need New Evaluation Metrics for NLGcited by 232
  9. 2022Revisiting the Gold Standard: Grounding Summarization Evaluation with Robust Human Evaluationcited by 141
  10. 2023WiCE: Real-World Entailment for Claims in Wikipediacited by 118
  11. 2023LongEval: Guidelines for Human Evaluation of Faithfulness in Long-form Summarizationcited by 96
  12. 2023Topical-Chat: Towards Knowledge-Grounded Open-Domain Conversationscited by 88
  13. 2019ACUTE-EVAL: Improved Dialogue Evaluation with Optimized Questions and Multi-turn Comparisonscited by 83
  14. 2023TIGERScore: Towards Building Explainable Metric for All Text Generation Taskscited by 69
  15. 2022Generating Literal and Implied Subquestions to Fact-check Complex Claimscited by 66
  16. 2022Generating Scientific Claims for Zero-Shot Scientific Fact Checkingcited by 59
  17. 2023Evaluating Evaluation Metrics: A Framework for Analyzing NLG Evaluation Metrics using Measurement Theorycited by 41
  18. 2024HD-Eval: Aligning Large Language Model Evaluators Through Hierarchical Criteria Decompositioncited by 33
  19. 2023X-Eval: Generalizable Multi-aspect Text Evaluation via Augmented Instruction Tuning with Auxiliary Evaluation Aspectscited by 28
  20. 2020Perception Score, A Learned Metric for Open-ended Text Generation Evaluationcited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 20; 2026: 63.

87 papers.

Sort:

By area:

  1. 2025TRAIL: Trace Reasoning and Agentic Issue Localizationcited by 98
  2. 2025Transforming Science with Large Language Models: A Survey on AI-assisted Scientific Discovery, Experimentation, Content Generation, and Evaluationcited by 41
  3. 2024LMUnit: Fine-grained Evaluation with Natural Language Unit Testscited by 32
  4. 2026RubricBench: Aligning Model-Generated Rubrics with Human Standardscited by 21
  5. 2025RExBench: Can coding agents autonomously implement AI research extensions?cited by 17
  6. 2024Training Language Models to Critique With Multi-agent Feedbackcited by 16
  7. 2025DocAgent: A Multi-Agent System for Automated Code Documentation Generationcited by 16
  8. 2024GLIDER: Grading LLM Interactions and Decisions using Explainable Rankingcited by 14
  9. 2024Think Together and Work Better: Combining Humans' and LLMs' Think-Aloud Outcomes for Effective Text Evaluationcited by 12
  10. 2025DEER: A Benchmark for Evaluating Deep Research Agents on Expert Report Generationcited by 12
  11. 2025A Scalable Framework for Evaluating Health Language Modelscited by 11
  12. 2025RECAST: Expanding the Boundaries of LLMs' Complex Instruction Following with Multi-Constraint Datacited by 11
38 more
  1. 2025FinTrust: A Comprehensive Benchmark of Trustworthiness Evaluation in Finance Domaincited by 8
  2. 2026SREGym: A Live Benchmark for AI SRE Agents with High-Fidelity Failure Scenarioscited by 8
  3. 2026Generating and Refining Dynamic Evaluation Rubrics for LLM-as-a-Judgecited by 7
  4. 2025Expert Preference-based Evaluation of Automated Related Work Generationcited by 6
  5. 2026From Rubrics to Reliable Scores: Evidence-Grounded Text Evaluation with LLM Judgescited by 6
  6. 2026Qworld: Question-Specific Evaluation Criteria for LLMscited by 6
  7. 2025A Multi-Task Evaluation of LLMs' Processing of Academic Text Inputcited by 5
  8. 2025Behavioral Fingerprinting of Large Language Modelscited by 5
  9. 2026AutoChecklist: Composable Pipelines for Checklist Generation and Scoring with LLM-as-a-Judgecited by 5
  10. 2026JEV vs. LLMs as Rubric Judges: Cheaper, Faster, and Wrong in the Same Placescited by 5
  11. 2025From Feedback to Checklists: Grounded Evaluation of AI-Generated Clinical Notescited by 4
  12. 2025Reference-Free Rating of LLM Responses via Latent Informationcited by 4
  13. 2025Evaluating Legal Reasoning Traces with Legal Issue Tree Rubricscited by 4
  14. 2026Beyond Single-shot Writing: Deep Research Agents are Unreliable at Multi-turn Report Revisioncited by 4
  15. 2026Safeguarding LLM Agents from Misalignment through Provenance Analysiscited by 4
  16. 2025Automated Refinement of Essay Scoring Rubrics for Language Models via Reflect-and-Revisecited by 3
  17. 2025A Principle-based Framework for the Development and Evaluation of Large Language Models for Health and Wellnesscited by 3
  18. 2026Beyond the Illusion of Consensus: From Surface Heuristics to Knowledge-Grounded Evaluation in LLM-as-a-Judgecited by 3
  19. 2026Cross-Lingual LLM-Judge Transfer via Evaluation Decompositioncited by 3
  20. 2026C2: Scalable Rubric-Augmented Reward Modeling from Binary Preferencescited by 3
  21. 2026RUBRIC-ARROW: Alternating Pointwise Rubric Reward Modeling for LLM Post-training in Non-verifiable Domainscited by 3
  22. 2026LLM Judges Can Be Too Generous When There Is No Reference Answercited by 3
  23. 2026FrontierFinance: A Challenging Benchmark for Measuring Frontier Intelligence of Finance Agentscited by 3
  24. 2025AURA: A Diagnostic Framework for Tracking User Satisfaction of Interactive Planning Agentscited by 2
  25. 2025EssayBench: Evaluating Large Language Models in Multi-Genre Chinese Essay Writingcited by 2
  26. 2025Reward Models are Metrics in a Trench Coatcited by 2
  27. 2025ARISE: Agentic Rubric-Guided Iterative Survey Engine for Automated Scholarly Paper Generationcited by 2
  28. 2026Developing and evaluating a chatbot to support maternal health carecited by 2
  29. 2026GAIA-v2-LILT: Multilingual Adaptation of Agent Benchmark beyond Translationcited by 2
  30. 2026DynSess: Dynamic Session-Level Evaluation and Optimization Framework for Role-Playing Agentscited by 2
  31. 2026Two-Level Meta-Rubrics for Evaluating Open-Ended Generation: GAMUT, a Benchmark for Factual Completenesscited by 2
  32. 2026PGMem: Tightly Coupled Persona-Memory Graph for Lifelong Personalized Agentscited by 2
  33. 2026Can LLM Agents Stick to the Script? A Benchmark for Long-Horizon Consistency in Interactive Narrativescited by 2
  34. 2026Doc-PP: Document Policy Preservation Benchmark for Large Vision-Language Modelscited by 1
  35. 2026MemoNoveltyAgent: A Historical Research Memory-Aware Agent Workflow for Paper Novelty Assessmentcited by 1
  36. 2026Adaptive Cost-Efficient Evaluation for Reliable Patent Claim Generationcited by 1
  37. 2026MLLM-as-a-Judge Exhibits Model Preference Biascited by 1
  38. 2026StratMem-Bench: Evaluating Strategic Memory Use in Virtual Character Conversation Beyond Factual Recallcited by 1

50 most cited of 87 All 87 (JSON)