mlsanityData to 7 Oct 2026

Towards Multilingual LLM Evaluation for European Languages

arXiv 2410.08928 ↗First posted 11 Oct 2024cs.CL cs.AI cs.LGcited by 29

Not on the map. Why?

Abstract (arXiv)

The rise of Large Language Models (LLMs) has revolutionized natural language processing across numerous languages and tasks. However, evaluating LLM performance in a consistent and meaningful way across multiple European languages remains challenging, especially due to the scarcity of language-parallel multilingual benchmarks. We introduce a multilingual evaluation approach tailored for European languages. We employ translated versions of five widely-used benchmarks to assess the capabilities of 40 LLMs across 21 European languages. Our contributions include examining the effectiveness of translated benchmarks, assessing the impact of different translation services, and offering a multilingual evaluation framework for LLMs that includes newly created datasets: EU20-MMLU, EU20-HellaSwag, EU20-ARC, EU20-TruthfulQA, and EU20-GSM8K. The benchmarks and results are made publicly available to encourage further research in multilingual LLM evaluation.

Key facts. Cited by 29 papers in the map (data through 7 Oct 2026). First posted on arXiv 11 Oct 2024.

Builds on

0 of 0 references.

No references here yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 12; 2026: 15.

29 papers.

Sort:

By area:

  1. 2024MM-Eval: A Multilingual Meta-Evaluation Benchmark for LLM-as-a-Judge and Reward Modelscited by 34
  2. 2025Global PIQA: Evaluating Commonsense Reasoning Across 100+ Languages and Culturescited by 8
  3. 2025The AI Language Proficiency Monitor -- Tracking the Progress of LLMs on Multilingual Benchmarkscited by 7
  4. 2024BEIR-NL: Zero-shot Information Retrieval Benchmark for the Dutch Languagecited by 6
  5. 2025Demystifying Multilingual Chain-of-Thought in Process Reward Modelingcited by 6
  6. 2025Checklist Engineering Empowers Multilingual LLM Judgescited by 3
  7. 2025MELAC: Massive Evaluation of Large Language Models with Alignment of Culture in Persian Languagecited by 3
  8. 2025LAG-MMLU: Benchmarking Frontier LLM Understanding in Latvian and Giriamacited by 2
  9. 2026UrduBench: An Urdu Reasoning Benchmark using Contextually Ensembled Translations with Human-in-the-Loopcited by 2
  10. 2025Towards Multilingual LLM Evaluation for Baltic and Nordic languages: A study on Lithuanian Historycited by 1
  11. 2025Are Multilingual Language Models an Off-ramp for Under-resourced Languages? Will we arrive at Digital Language Equality in Europe in 2030?cited by 1
  12. 2025Multilingual European Language Models: Benchmarking Approaches and Challengescited by 1
17 more
  1. 2025Georeferencing complex relative locality descriptions with large language modelscited by 1
  2. 2026Racka: Efficient Hungarian LLM Adaptation on Academic Infrastructurecited by 1
  3. 2026EstLLM: Enhancing Estonian Capabilities in Multilingual LLMs via Continued Pretraining and Post-Trainingcited by 1
  4. 2026TildeOpen LLM: Leveraging Curriculum Learning to Achieve Equitable Language Representationcited by 1
  5. 2026Quantifying the Impact of Translation Errors on Multilingual LLM Evaluationcited by 1
  6. 2026CARTE: A Benchmark for Mapping Language Model Knowledge Across Francecited by 1
  7. 2025Enhancing Traffic Accident Classifications: Application of NLP Methods for City Safetycited by 0
  8. 2025Estonian WinoGrande Dataset: Comparative Analysis of LLM Performance on Human and Machine Translationcited by 0
  9. 2026textitVersteasch du mi? Computational and Socio-Linguistic Perspectives on GenAI, LLMs, and Non-Standard Languagecited by 0
  10. 2026Diagnosing Translated Benchmarks: An Automated Quality Assurance Study of the EU20 Benchmark Suitecited by 0
  11. 2026RespondeoQA: a Benchmark for Bilingual Latin-English Question Answeringcited by 0
  12. 2026Beyond English: Uncovering the Multilingual Gap in Vision-Language-Action Modelscited by 0
  13. 2026Meta-Learning Preferences for Multilingual LLM Alignmentcited by 0
  14. 2026From Data to Device: ELMOD An Efficient German-First 2.7B Language Model for Mobile Inferencecited by 0
  15. 2026Language Equality has a Price: A Systematic Investigation of Multi-turn LLM Performance for EU-24+cited by 0
  16. 2026To Each Language Its Tokenizer: Modular Tokenizers for Efficient Multilingual LLMscited by 0
  17. 2026Made in Hungary: Comments on the performance of generative language modelscited by 0