mlsanityData to 7 Oct 2026

How Reliable is Multilingual LLM-as-a-Judge?

arXiv 2505.12201 ↗First posted 18 May 2025cs.CLcited by 29

Not on the map. Why?

Abstract (arXiv)

LLM-as-a-Judge has emerged as a popular evaluation strategy, where advanced large language models assess generation results in alignment with human instructions. While these models serve as a promising alternative to human annotators, their reliability in multilingual evaluation remains uncertain. To bridge this gap, we conduct a comprehensive analysis of multilingual LLM-as-a-Judge. Specifically, we evaluate five models from different model families across five diverse tasks involving 25 languages. Our findings reveal that LLMs struggle to achieve consistent judgment results across languages, with an average Fleiss' Kappa of approximately 0.3, and some models performing even worse. To investigate the cause of inconsistency, we analyze various influencing factors. We observe that consistency varies significantly across languages, with particularly poor performance in low-resource languages. Additionally, we find that neither training on multilingual data nor increasing model scale directly improves judgment consistency. These findings suggest that LLMs are not yet reliable for evaluating multilingual predictions. We finally propose an ensemble strategy which improves the consistency of the multilingual judge in real-world applications.

Key facts. Cited by 29 papers in the map (data through 7 Oct 2026). Builds on 30 of its 47 references that are in the map. First posted on arXiv 18 May 2025.

Builds on

30 of 47 references.

References · 30

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2024The Llama 3 Herd of Modelscited by 14,192
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  5. 2024A Survey on LLM-as-a-Judgecited by 1,384
  6. 2020BLEURT: Learning Robust Metrics for Text Generationcited by 1,196
  7. 2023Large Language Models are not Fair Evaluatorscited by 1,050
  8. 2023Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formattingcited by 778
  9. 2023Can Large Language Models Be an Alternative to Human Evaluations?cited by 729
  10. 2021BARTScore: Evaluating Generated Text as Text Generationcited by 719
  11. 2019On the Cross-lingual Transferability of Monolingual Representationscited by 575
  12. 2023Prometheus: Inducing Fine-grained Evaluation Capability in Language Modelscited by 558
18 more
  1. 2024From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judgecited by 480
  2. 2023Is ChatGPT a Good NLG Evaluator? A Preliminary Studycited by 420
  3. 2023Large Language Models Are State-of-the-Art Evaluators of Translation Qualitycited by 416
  4. 2024Justice or Prejudice? Quantifying Biases in LLM-as-a-Judgecited by 354
  5. 2023JudgeLM: Fine-tuned Large Language Models are Scalable Judgescited by 351
  6. 2024Replacing Judges with Juries: Evaluating LLM Generations with a Panel of Diverse Modelscited by 301
  7. 2023MEGA: Multilingual Evaluation of Generative AIcited by 291
  8. 2024Meta-Rewarding Language Models: Self-Improving Alignment with LLM-as-a-Meta-Judgecited by 202
  9. 2020A Survey of Evaluation Metrics Used for NLG Systemscited by 164
  10. 2020WikiLingua: A New Benchmark Dataset for Cross-Lingual Abstractive Summarizationcited by 154
  11. 2023Human-like Summarization Evaluation with ChatGPTcited by 126
  12. 2024Aya Expanse: Combining Research Breakthroughs for a New Multilingual Frontiercited by 121
  13. 2024Is LLM-as-a-Judge Robust? Investigating Universal Adversarial Attacks on Zero-shot LLM Assessmentcited by 117
  14. 2023The Devil is in the Errors: Leveraging Large Language Models for Fine-grained Machine Translation Evaluationcited by 95
  15. 2024OffsetBias: Leveraging Debiased Data for Tuning Evaluatorscited by 94
  16. 2023Are Large Language Model-based Evaluators the Solution to Scaling Up Multilingual Evaluation?cited by 87
  17. 2024PARIKSHA: A Large-Scale Investigation of Human-LLM Evaluator Agreement on Multilingual and Multi-Cultural Datacited by 37
  18. 2024Finding Blind Spots in Evaluator LLMs with Interpretable Checklistscited by 32

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 5; 2026: 24.

29 papers.

Sort:

By area:

  1. 2026Reliability without Validity: A Systematic, Large-Scale Evaluation of LLM-as-a-Judge Models Across Agreement, Consistency, and Biascited by 19
  2. 2026From RAG to Agentic RAG for Faithful Islamic Question Answeringcited by 9
  3. 2025Cetvel: A Unified Benchmark for Evaluating Language Understanding, Generation and Cultural Capacity of LLMs for Turkishcited by 7
  4. 2026ChartEditBench: Evaluating Grounded Multi-Turn Chart Editing in Multimodal Language Modelscited by 6
  5. 2026propella-1: Multi-Property Document Annotation for LLM Data Curation at Scalecited by 5
  6. 2025Checklist Engineering Empowers Multilingual LLM Judgescited by 3
  7. 2025MENLO: From Preferences to Proficiency -- Evaluating and Modeling Native-like Quality Across 47 Languagescited by 3
  8. 2026Fairness or Fluency? An Investigation into Language Bias of Pairwise LLM-as-a-Judgecited by 3
  9. 2026Cross-Lingual LLM-Judge Transfer via Evaluation Decompositioncited by 3
  10. 2026Beyond LLM-as-a-Judge: Deterministic Metrics for Multilingual Generative Text Evaluationcited by 3
  11. 2026MTVA-Bench: Evaluating the Language Model Inside Cascaded Voice Agentscited by 3
  12. 2026Mitigating Translationese Bias in Multilingual LLM-as-a-Judge via Disentangled Information Bottleneckcited by 2
17 more
  1. 2026When Languages Disagree: Self-Evolving Multilingual LLM Judgescited by 2
  2. 2026JudgeArena: A Unified Framework for Reproducible LLM-Judge Evaluationcited by 2
  3. 2026Multilingual Prompt Localization for Agent-as-a-Judge: Language and Backbone Sensitivity in Requirement-Level Evaluationcited by 1
  4. 2026Benchmarking Frontier LLMs on Arabic Cultural and Sociolinguistic Knowledge: A Cross-Evaluation Framework with Human SME Ground Truthcited by 1
  5. 2026Challenges and Recommendations for LLM-as-a-Judge in Multilingual Settings and for Low-Resource Languagescited by 1
  6. 2025M-DaQ: Retrieving Samples with Multilingual Diversity and Quality for Instruction Fine-Tuning Datasetscited by 0
  7. 2025Measuring Moral LLM Responses in Multilingual Capacitiescited by 0
  8. 2026Cross-Lingual Stability of LLM Judges Under Controlled Generation: Evidence from Finno-Ugric Languagescited by 0
  9. 2026CAKE: Cloud Architecture Knowledge Evaluation of Large Language Modelscited by 0
  10. 2026Evaluating Cross-lingual Knowledge Consistency in Code-Mixed vis-a-vis Indian Languages using IndicKLARcited by 0
  11. 2026LLM Judges Inconsistently Disagree Across Safety Criteria and Harm Categoriescited by 0
  12. 2026SPLIT: Cross-Lingual Empathy and Cultural Grounding in English and Ukrainian LLM Responsescited by 0
  13. 2026Lower-Resource, Higher Scores: Language Bias in LLM Evaluatorscited by 0
  14. 2026Multilingual Verifier Bias in RLVR: Benchmark, Rollout Diagnosis, and the Cross-Lingual Selection Bottleneckcited by 0
  15. 2026Rank Reversal in Multilingual LLM Judges: A Label-Free Double-Centering Calibratorcited by 0
  16. 2026How AI Assistants Respond to Repeated Abusecited by 0
  17. 2026V=akQA: A Benchmark and Evaluation Study for Telugu Spoken Factoid Question Answeringcited by 0