mlsanityData to 14 Oct 2026

Are Large Language Model-based Evaluators the Solution to Scaling Up Multilingual Evaluation?

arXiv 2309.07462 ↗First posted 14 Sep 2023cs.CLcited by 88

Not on the map. Why?

Abstract (arXiv)

Large Language Models (LLMs) excel in various Natural Language Processing (NLP) tasks, yet their evaluation, particularly in languages beyond the top 20, remains inadequate due to existing benchmarks and metrics limitations. Employing LLMs as evaluators to rank or score other models' outputs emerges as a viable solution, addressing the constraints tied to human annotators and established benchmarks. In this study, we explore the potential of LLM-based evaluators, specifically GPT-4 in enhancing multilingual evaluation by calibrating them against 20K human judgments across three text-generation tasks, five metrics, and eight languages. Our analysis reveals a bias in GPT4-based evaluators towards higher scores, underscoring the necessity of calibration with native speaker judgments, especially in low-resource and non-Latin script languages, to ensure accurate evaluation of LLM performance across diverse languages.

Key facts. Cited by 88 papers in the map (data through 14 Oct 2026). Builds on 26 of its 47 references that are in the map. First posted on arXiv 14 Sep 2023.

Builds on

26 of 47 references.

References · 26

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  4. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
  5. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,442
  6. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,050
  7. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,276
  8. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,434
  9. 2023A Survey on Evaluation of Large Language Modelscited by 1,242
  10. 2023PaLM 2 Technical Reportcited by 1,162
  11. 2023Large Language Models are not Fair Evaluatorscited by 1,085
  12. 2023ChatGPT Outperforms Crowd-Workers for Text-Annotation Taskscited by 870
14 more
  1. 2020The State and Fate of Linguistic Diversity and Inclusion in the NLP Worldcited by 798
  2. 2023ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debatecited by 742
  3. 2023Can Large Language Models Be an Alternative to Human Evaluations?cited by 734
  4. 2023Is ChatGPT a Good NLG Evaluator? A Preliminary Studycited by 422
  5. 2023Large Language Models Are State-of-the-Art Evaluators of Translation Qualitycited by 420
  6. 2023GPTScore: Evaluate as You Desirecited by 361
  7. 2023MEGA: Multilingual Evaluation of Generative AIcited by 293
  8. 2023Stop Uploading Test Data in Plain Text: Practical Strategies for Mitigating Data Contamination by Evaluation Benchmarkscited by 142
  9. 2023The Devil is in the Errors: Leveraging Large Language Models for Fine-grained Machine Translation Evaluationcited by 95
  10. 2023Wider and Deeper LLM Networks are Fairer LLM Evaluatorscited by 94
  11. 2023Automated Annotation with Generative AI Requires Validationcited by 76
  12. 2023GPTEval: A Survey on Assessments of ChatGPT and GPT-4cited by 73
  13. 2022Beyond English-Centric Bitexts for Better Multilingual Language Representation Learningcited by 16
  14. 2022Beyond Static Models and Test Sets: Benchmarking the Potential of Pre-trained Models Across Tasks and Languagescited by 10

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 5; 2024: 41; 2025: 28; 2026: 14.

88 papers.

Sort:

By area:

  1. 2024From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judgecited by 494
  2. 2024LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methodscited by 465
  3. 2024LLMs instead of Human Judges? A Large Scale Empirical Study across 20 NLP Evaluation Taskscited by 243
  4. 2024Judging the Judges: Evaluating Alignment and Vulnerabilities in LLMs-as-Judgescited by 205
  5. 2023Branch-Solve-Merge Improves Large Language Model Evaluation and Generationcited by 106
  6. 2024The Butterfly Effect of Altering Prompts: How Small Changes and Jailbreaks Affect Large Language Model Performancecited by 100
  7. 2024A Systematic Survey and Critical Review on Evaluating Large Language Models: Challenges, Limitations, and Recommendationscited by 85
  8. 2024Towards Conversational Diagnostic AIcited by 79
  9. 2024LLM-based NLG Evaluation: Current Status and Challengescited by 73
  10. 2023Split and Merge: Aligning Position Biases in LLM-based Evaluatorscited by 72
  11. 2024MoRAL: MoE Augmented LoRA for LLMs' Lifelong Learningcited by 63
  12. 2024Multilingual Large Language Model: A Survey of Resources, Taxonomy and Frontierscited by 51
38 more
  1. 2024Financial Report Chunking for Effective Retrieval Augmented Generationcited by 46
  2. 2024Auto-Arena: Automating LLM Evaluations with Agent Peer Battles and Committee Discussionscited by 42
  3. 2023Turning English-centric LLMs Into Polyglots: How Much Multilinguality Is Needed?cited by 39
  4. 2024PARIKSHA: A Large-Scale Investigation of Human-LLM Evaluator Agreement on Multilingual and Multi-Cultural Datacited by 37
  5. 2024Are LLM-based Evaluators Confusing NLG Quality Criteria?cited by 36
  6. 2024RTP-LX: Can LLMs Evaluate Toxicity in Multilingual Scenarios?cited by 36
  7. 2024MM-Eval: A Multilingual Meta-Evaluation Benchmark for LLM-as-a-Judge and Reward Modelscited by 35
  8. 2024HD-Eval: Aligning Large Language Model Evaluators Through Hierarchical Criteria Decompositioncited by 33
  9. 2024CaLMQA: Exploring culturally specific long-form question answering across 23 languagescited by 31
  10. 2025The Bitter Lesson Learned from 2,000+ Multilingual Benchmarkscited by 31
  11. 2025How Reliable is Multilingual LLM-as-a-Judge?cited by 30
  12. 2024Language Imbalance Driven Rewarding for Multilingual Self-improvingcited by 28
  13. 2024Reference-Guided Verdict: LLMs-as-Judges in Automatic Evaluation of Free-Form QAcited by 25
  14. 2025An Empirical Analysis of Uncertainty in Large Language Model Evaluationscited by 20
  15. 2024Evaluating the Performance of Large Language Models via Debatescited by 19
  16. 2023DEsignBench: Exploring and Benchmarking DALL-E 3 for Imagining Visual Designcited by 17
  17. 2024METAL: Towards Multilingual Meta-Evaluationcited by 15
  18. 2024How Reliable Are Automatic Evaluation Methods for Instruction-Tuned LLMs?cited by 13
  19. 2024Machine Translation Hallucination Detection for Low and High Resource Languages using Large Language Modelscited by 13
  20. 2025SeaExam and SeaBench: Benchmarking LLMs with Local Multilingual Questions in Southeast Asiacited by 13
  21. 2025XIFBench: Evaluating Large Language Models on Multilingual Instruction Followingcited by 11
  22. 2025Do LLMs Understand Your Translations? Evaluating Paragraph-level MT with Question Answeringcited by 10
  23. 2024'Since Lawyers are Males..': Examining Implicit Gender Bias in Hindi Language Generation by LLMscited by 8
  24. 2025Do You Know About My Nation? Investigating Multilingual Language Models' Cultural Literacy Through Factual Knowledgecited by 8
  25. 2024Beyond Metrics: Evaluating LLMs' Effectiveness in Culturally Nuanced, Low-Resource Real-World Scenarioscited by 7
  26. 2024Can visual language models resolve textual ambiguity with visual cues? Let visual puns tell you!cited by 7
  27. 2025CLIMATEAGENT: Multi-Agent Orchestration for Complex Climate Data Science Workflowscited by 7
  28. 2025CLEV: LLM-Based Evaluation Through Lightweight Efficient Voting for Free-Form Question-Answeringcited by 6
  29. 2025A Multilingual, Culture-First Approach to Addressing Misgendering in LLM Applicationscited by 6
  30. 2026Translation as a Scalable Proxy for Multilingual Evaluationcited by 6
  31. 2024High-quality Data-to-Text Generation for Severely Under-Resourced Languages with Out-of-the-box Large Language Modelscited by 5
  32. 2024Aggregation of Reasoning: A Hierarchical Framework for Enhancing Answer Selection in Large Language Modelscited by 5
  33. 2024On the Evaluation Practices in Multilingual NLP: Can Machine Translation Offer an Alternative to Human Translations?cited by 5
  34. 2024Awes, Laws, and Flaws From Today's LLM Researchcited by 5
  35. 2024LLMs are Biased Evaluators But Not Biased for Retrieval Augmented Generationcited by 5
  36. 2024Domain-adaptative Continual Learning for Low-resource Tasks: Evaluation on Nepalicited by 5
  37. 2025Exploring the Multilingual NLG Evaluation Abilities of LLM-Based Evaluatorscited by 5
  38. 2025The Thin Line Between Comprehension and Persuasion in LLMscited by 5

50 most cited of 88 All 88 (JSON)