mlsanityData to 7 Oct 2026

Challenges and Recommendations for LLM-as-a-Judge in Multilingual Settings and for Low-Resource Languages

arXiv 2607.02235 ↗First posted 2 Jul 2026cs.CL cs.AIcited by 1

Not on the map. Why?

Abstract (arXiv)

LLM-as-a-Judge has become the dominant evaluation paradigm for many natural language generation tasks (albeit mostly in English) due to shortcomings of conventional metrics and high correlations with human judgment. There are now attempts to extend LLM-as-a-Judge to multilingual settings including low-resource languages. However, LLMs have limited proficiency in low-resource languages, and there is often no adequate human validation in these settings. To highlight the scope of the problem and current practices, we explore the use of LLM-as-a-Judge evaluators in ACL Anthology papers focusing on multilingual settings and low-resource languages across a diverse set of tasks. Out of 650 papers mentioning LLM-as-a-judge, only 33 of them focus on low-resource or multilingual settings. Our in-depth analysis of these papers indicates inconsistent evaluation outcomes, a tendency to overtrust LLM judgments in multilingual settings, and the widespread reliance on a single judge model per study. To help the NLP community further, we conclude with a checklist of recommendations about how to use LLM-as-a-Judge in multilingual and low-resource settings

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 54 of its 90 references that are in the map. First posted on arXiv 2 Jul 2026.

Builds on

54 of 90 references.

References · 54

  1. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  2. 2024A Survey on LLM-as-a-Judgecited by 1,384
  3. 2020The State and Fate of Linguistic Diversity and Inclusion in the NLP Worldcited by 789
  4. 2024LLM Evaluators Recognize and Favor Their Own Generationscited by 651
  5. 2024From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judgecited by 480
  6. 2024LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methodscited by 451
  7. 2024MLLM-as-a-Judge: Assessing Multimodal LLM-as-a-Judge with Vision-Language Benchmarkcited by 362
  8. 2024Humans or LLMs as the Judge? A Study on Judgement Biasescited by 314
  9. 2024LLMs instead of Human Judges? A Large Scale Empirical Study across 20 NLP Evaluation Taskscited by 236
  10. 2023Evaluating Correctness and Faithfulness of Instruction-Following Models for Question Answeringcited by 150
  11. 2025ELEPHANT: Measuring and understanding social sycophancy in LLMscited by 115
  12. 2023LLMs as Narcissistic Evaluators: When Ego Inflates Evaluation Scorescited by 88
38 more
  1. 2023Are Large Language Model-based Evaluators the Solution to Scaling Up Multilingual Evaluation?cited by 87
  2. 2024Ethical Reasoning and Moral Value Alignment of LLMs Depend on the Language we Prompt them incited by 45
  3. 2023AfroBench: How Good are Large Language Models on African Languages?cited by 43
  4. 2024How Does Quantization Affect Multilingual LLMs?cited by 41
  5. 2024PARIKSHA: A Large-Scale Investigation of Human-LLM Evaluator Agreement on Multilingual and Multi-Cultural Datacited by 37
  6. 2025Neither Valid nor Reliable? Investigating the Use of LLMs as Judgescited by 37
  7. 2024MM-Eval: A Multilingual Meta-Evaluation Benchmark for LLM-as-a-Judge and Reward Modelscited by 34
  8. 2025How Reliable is Multilingual LLM-as-a-Judge?cited by 29
  9. 2024Constructing Domain-Specific Evaluation Sets for LLM-as-a-judgecited by 28
  10. 2024MQM-APE: Toward High-Quality Error Annotation Predictors with Automatic Post-Editing in LLM Translation Evaluatorscited by 22
  11. 2024Enhancing Text Classification through LLM-Driven Active Learning and Human Annotationcited by 20
  12. 2024Cross-Lingual Auto Evaluation for Assessing Multilingual LLMscited by 17
  13. 2023Representativeness as a Forgotten Lesson for Multilingual and Code-switched Data Collection and Preparationcited by 16
  14. 2024SciEx: Benchmarking Large Language Models on Scientific Exams with Human Expert Grading and Automatic Gradingcited by 14
  15. 2024MIRAGE-Bench: Automatic Multilingual Benchmark Arena for Retrieval-Augmented Generation Systemscited by 14
  16. 2023MELA: Multilingual Evaluation of Linguistic Acceptabilitycited by 13
  17. 2025X-Guard: Multilingual Guard Agent for Content Moderationcited by 13
  18. 2023"Knowing When You Don't Know": A Multilingual Relevance Assessment Dataset for Robust Retrieval-Augmented Generationcited by 12
  19. 2025Evaluating the Quality of Benchmark Datasets for Low-Resource Languages: A Case Study on Turkishcited by 12
  20. 2024Dialectal Toxicity Detection: Evaluating LLM-as-a-Judge Consistency Across Language Varietiescited by 10
  21. 2025SSA-COMET: Do LLMs Outperform Learned Metrics in Evaluating MT for Under-Resourced African Languages?cited by 10
  22. 2025Do we still need Human Annotators? Prompting Large Language Models for Aspect Sentiment Quad Predictioncited by 9
  23. 2023Analyzing Multilingual Competency of LLMs in Multi-Turn Instruction Following: A Case Study of Arabiccited by 7
  24. 2024JumpCoder: Go Beyond Autoregressive Coder via Online Modificationcited by 7
  25. 2024Socially Responsible Data for Large Multilingual Language Modelscited by 7
  26. 2025MEMERAG: A Multilingual End-to-End Meta-Evaluation Benchmark for Retrieval Augmented Generationcited by 7
  27. 2024Unlocking Legal Knowledge: A Multilingual Dataset for Judicial Summarization in Switzerlandcited by 6
  28. 2025A Multilingual, Culture-First Approach to Addressing Misgendering in LLM Applicationscited by 6
  29. 2024Improving Language Models Trained on Translated Data with Continual Pre-Training and Dictionary Learning Analysiscited by 5
  30. 2024ECoh: Turn-level Coherence Evaluation for Multilingual Dialoguescited by 5
  31. 2024Domain-adaptative Continual Learning for Low-resource Tasks: Evaluation on Nepalicited by 5
  32. 2024Comparing LLM prompting with Cross-lingual transfer performance on Indigenous and Low-resource Brazilian Languagescited by 4
  33. 2024Estimating the Level of Dialectness Predicts Interannotator Agreement in Multi-dialect Arabic Datasetscited by 4
  34. 2025Evaluating Text Style Transfer Evaluation: Are There Any Reliable Metrics?cited by 4
  35. 2025Dr.Copilot: A Multi-Agent Prompt Optimized Assistant for Improving Patient-Doctor Communication in Romaniancited by 4
  36. 2025Translate, then Detect: Leveraging Machine Translation for Cross-Lingual Toxicity Classificationcited by 4
  37. 2026LALM-as-a-Judge: Benchmarking Large Audio-Language Models for Safety Evaluation in Multi-Turn Spoken Dialoguescited by 4
  38. 2025It Depends: Resolving Referential Ambiguity in Minimal Contexts with Commonsense Knowledgecited by 3

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026When Do Agents Help? Embedding, LLM and Agentic Alignment of Classical Texts and Their Translationscited by 0