mlsanityData to 14 Oct 2026

Checklist Engineering Empowers Multilingual LLM Judges

arXiv 2507.06774 ↗First posted 9 Jul 2025cs.CLcited by 3

Not on the map. Why?

Abstract (arXiv)

Automated text evaluation has long been a central issue in Natural Language Processing (NLP). Recently, the field has shifted toward using Large Language Models (LLMs) as evaluators-a trend known as the LLM-as-a-Judge paradigm. While promising and easily adaptable across tasks, this approach has seen limited exploration in multilingual contexts. Existing multilingual studies often rely on proprietary models or require extensive training data for fine-tuning, raising concerns about cost, time, and efficiency. In this paper, we propose Checklist Engineering based LLM-as-a-Judge (CE-Judge), a training-free framework that uses checklist intuition for multilingual evaluation with an open-source model. Experiments across multiple languages and three benchmark datasets, under both pointwise and pairwise settings, show that our method generally surpasses the baselines and performs on par with the GPT-4o model.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 21 of its 29 references that are in the map. First posted on arXiv 9 Jul 2025.

Builds on

21 of 29 references.

References · 21

  1. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  2. 2024Qwen2.5 Technical Reportcited by 4,942
  3. 2024A Survey on LLM-as-a-Judgecited by 1,423
  4. 2024From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judgecited by 494
  5. 2024Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Modelscited by 446
  6. 2024LLMs instead of Human Judges? A Large Scale Empirical Study across 20 NLP Evaluation Taskscited by 243
  7. 2024TICKing All the Boxes: Generated Checklists Improve LLM Evaluation and Generationcited by 68
  8. 2024M-RewardBench: Evaluating Reward Models in Multilingual Settingscited by 50
  9. 2025RocketEval: Efficient Automated LLM Evaluation via Grading Checklistcited by 39
  10. 2025M-Prometheus: A Suite of Open Multilingual LLM Judgescited by 38
  11. 2024MM-Eval: A Multilingual Meta-Evaluation Benchmark for LLM-as-a-Judge and Reward Modelscited by 35
  12. 2024Finding Blind Spots in Evaluator LLMs with Interpretable Checklistscited by 32
9 more
  1. 2024Towards Multilingual LLM Evaluation for European Languagescited by 30
  2. 2025How Reliable is Multilingual LLM-as-a-Judge?cited by 30
  3. 2024How Good Are LLMs for Literary Translation, Really? Literary Translation Evaluation with Humans and LLMscited by 23
  4. 2025Beyond English: The Impact of Prompt Translation Strategies across Languages and Tasks in Multilingual LLMscited by 22
  5. 2024Cross-Lingual Auto Evaluation for Assessing Multilingual LLMscited by 18
  6. 2024LLM-as-an-Interviewer: Beyond Static Testing Through Dynamic LLM Evaluationcited by 13
  7. 2024DnA-Eval: Enhancing Large Language Model Evaluation through Decomposition and Aggregationcited by 12
  8. 2024RepEval: Effective Text Evaluation with LLM Representationcited by 6
  9. 2025Exploring the Multilingual NLG Evaluation Abilities of LLM-Based Evaluatorscited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026Code Change Characteristics and Description Alignment: A Comparative Study of Agentic versus Human Pull Requestscited by 3
  2. 2026Beyond Time Shifts: Adapting Omni-LLM as a Reference-Free Evaluator for Generative Audio-Visual Modelscited by 1
  3. 2026LLJ Cards: Best practices for the Use of LLMs as Judgescited by 0