mlsanityData to 14 Oct 2026

Exploring the Multilingual NLG Evaluation Abilities of LLM-Based Evaluators

arXiv 2503.04360 ↗First posted 6 Mar 2025cs.CLcited by 5

Not on the map. Why?

Abstract (arXiv)

Previous research has shown that LLMs have potential in multilingual NLG evaluation tasks. However, existing research has not fully explored the differences in the evaluation capabilities of LLMs across different languages. To this end, this study provides a comprehensive analysis of the multilingual evaluation performance of 10 recent LLMs, spanning high-resource and low-resource languages through correlation analysis, perturbation attacks, and fine-tuning. We found that 1) excluding the reference answer from the prompt and using large-parameter LLM-based evaluators leads to better performance across various languages; 2) most LLM-based evaluators show a higher correlation with human judgments in high-resource languages than in low-resource languages; 3) in the languages where they are most sensitive to such attacks, they also tend to exhibit the highest correlation with human judgments; and 4) fine-tuning with data from a particular language yields a broadly consistent enhancement in the model's evaluation performance across diverse languages. Our findings highlight the imbalance in LLMs'evaluation capabilities across different languages and suggest that low-resource language scenarios deserve more attention.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 25 of its 69 references that are in the map. First posted on arXiv 6 Mar 2025.

Builds on

25 of 69 references.

References · 25

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  3. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  4. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  5. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,276
  6. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,998
  7. 2020BLEURT: Learning Robust Metrics for Text Generationcited by 1,203
  8. 2024ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Toolscited by 1,121
  9. 2020COMET: A Neural Framework for MT Evaluationcited by 1,033
  10. 2021XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scalecited by 601
  11. 2019Evaluating the Factual Consistency of Abstractive Text Summarizationcited by 584
  12. 2023Large Language Models Are State-of-the-Art Evaluators of Translation Qualitycited by 420
13 more
  1. 2020XCOPA: A Multilingual Dataset for Causal Commonsense Reasoningcited by 333
  2. 2023xCOMET: Transparent Machine Translation Evaluation through Fine-grained Error Detectioncited by 265
  3. 2023ChatGPT Beyond English: Towards a Comprehensive Evaluation of Large Language Models in Multilingual Learningcited by 206
  4. 2023Human-like Summarization Evaluation with ChatGPTcited by 126
  5. 2023Are Large Language Model-based Evaluators the Solution to Scaling Up Multilingual Evaluation?cited by 88
  6. 2024IndicGenBench: A Multilingual Benchmark to Evaluate Generation Capabilities of LLMs on Indic Languagescited by 56
  7. 2023SEAHORSE: A Multilingual, Multifaceted Dataset for Summarization Evaluationcited by 42
  8. 2024Can Large Language Models be Trusted for Evaluation? Scalable Meta-Evaluation of LLMs as Evaluators via Agent Debatecited by 41
  9. 2024Are LLM-based Evaluators Confusing NLG Quality Criteria?cited by 36
  10. 2021MTG: A Benchmark Suite for Multilingual Text Generationcited by 22
  11. 2023Simple LLM Prompting is State-of-the-Art for Robust and Multilingual Dialogue Evaluationcited by 19
  12. 2024Cross-Lingual Auto Evaluation for Assessing Multilingual LLMscited by 18
  13. 2024Large Language Models Are Active Critics in NLG Evaluationcited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 3; 2026: 2.

5 papers.

  1. 2025Checklist Engineering Empowers Multilingual LLM Judgescited by 3
  2. 2026Cross-Lingual LLM-Judge Transfer via Evaluation Decompositioncited by 3
  3. 2025PerQ: Efficient Evaluation of Multilingual Text Personalization Qualitycited by 0
  4. 2025SRS-Stories: Vocabulary-constrained multilingual story generation for language learningcited by 0
  5. 2026LLM Probe: Evaluating LLMs for Low-Resource Languagescited by 0