mlsanityData to 7 Oct 2026

Testing Low-Resource Language Support in LLMs Using Language Proficiency Exams: the Case of Luxembourgish

arXiv 2504.01667 ↗First posted 2 Apr 2025cs.CLcited by 2

Not on the map. Why?

Abstract (arXiv)

Large Language Models (LLMs) have become an increasingly important tool in research and society at large. While LLMs are regularly used all over the world by experts and lay-people alike, they are predominantly developed with English-speaking users in mind, performing well in English and other wide-spread languages while less-resourced languages such as Luxembourgish are seen as a lower priority. This lack of attention is also reflected in the sparsity of available evaluation tools and datasets. In this study, we investigate the viability of language proficiency exams as such evaluation tools for the Luxembourgish language. We find that large models such as Claude and DeepSeek-R1 typically achieve high scores, while smaller models show weak performances. We also find that the performances in such language exams can be used to predict performances in other NLP tasks in Luxembourgish.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 16 of its 51 references that are in the map. First posted on arXiv 2 Apr 2025.

Builds on

16 of 51 references.

References · 16

  1. 2024The Llama 3 Herd of Modelscited by 14,192
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  4. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  5. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,883
  6. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,116
  7. 2024ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Toolscited by 1,096
  8. 2024EuroLLM: Multilingual Language Models for Europecited by 103
  9. 2023Large Language Models Only Pass Primary School Exams in Indonesia: A Comprehensive Test on IndoMMLUcited by 53
  10. 2024SciEx: Benchmarking Large Language Models on Scientific Exams with Human Expert Grading and Automatic Gradingcited by 14
  11. 2024Text Generation Models for Luxembourgish with Limited Data: A Balanced Multilingual Strategycited by 7
  12. 2025LLMzSzL: a comprehensive LLM benchmark for Polishcited by 7
4 more
  1. 2024Disce aut Deficere: Evaluating LLMs Proficiency on the INVALSI Italian Benchmarkcited by 5
  2. 2024Cracking the Code: Multi-domain LLM Evaluation on Real-World Professional Exams in Indonesiacited by 4
  3. 2024Evaluating Large Language Models with Tests of Spanish as a Foreign Language: Pass or Fail?cited by 3
  4. 2024Examining the Behavior of LLM Architectures Within the Framework of Standardized National Exams in Brazilcited by 2

Built on by

Cited by year

Citing papers per year: 2025: 2.

2 papers.

  1. 2025MENLO: From Preferences to Proficiency -- Evaluating and Modeling Native-like Quality Across 47 Languagescited by 3
  2. 2025LuxIT: A Luxembourgish Instruction Tuning Dataset from Monolingual Seed Datacited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025MMLU-ProX: A Multilingual Benchmark for Advanced Large Language Model Evaluation
  2. 2023Through the Lens of Core Competency: Survey on Evaluation of Large Language Models
  3. 2024LLMs Beyond English: Scaling the Multilingual Capability of LLMs with Cross-Lingual Feedback
  4. 2025Neither Valid nor Reliable? Investigating the Use of LLMs as Judges
  5. 2024Enhancing LLM Evaluations: The Garbling Trick
  6. 2025Assessment of L2 Oral Proficiency using Speech Large Language Models
  7. 2026Lower-Resource, Higher Scores: Language Bias in LLM Evaluators
  8. 2024A Survey of Large Language Models for European Languages