mlsanityData to 14 Oct 2026

The AI Language Proficiency Monitor -- Tracking the Progress of LLMs on Multilingual Benchmarks

arXiv 2507.08538 ↗First posted 11 Jul 2025cs.CLcited by 7

Not on the map. Why?

Abstract (arXiv)

To ensure equitable access to the benefits of large language models (LLMs), it is essential to evaluate their capabilities across the world's languages. We introduce the AI Language Proficiency Monitor, a comprehensive multilingual benchmark that systematically assesses LLM performance across up to 200 languages, with a particular focus on low-resource languages. Our benchmark aggregates diverse tasks including translation, question answering, math, and reasoning, using datasets such as FLORES+, MMLU, GSM8K, TruthfulQA, and ARC. We provide an open-source, auto-updating leaderboard and dashboard that supports researchers, developers, and policymakers in identifying strengths and gaps in model performance. In addition to ranking models, the platform offers descriptive insights such as a global proficiency map and trends over time. By complementing and extending prior multilingual benchmarks, our work aims to foster transparency, inclusivity, and progress in multilingual AI. The system is available at https://huggingface.co/spaces/fair-forward/evals-for-every-language.

Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Builds on 23 of its 33 references that are in the map. First posted on arXiv 11 Jul 2025.

Builds on

23 of 33 references.

References · 23

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  3. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  4. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  5. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  6. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  7. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
  8. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  9. 2020AdapterFusion: Non-Destructive Task Composition for Transfer Learningcited by 788
  10. 2021The FLORES-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translationcited by 560
  11. 2022Language Models are Multilingual Chain-of-Thought Reasonerscited by 530
  12. 2018Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyondcited by 434
11 more
  1. 2020How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Modelscited by 318
  2. 2023MEGA: Multilingual Evaluation of Generative AIcited by 293
  3. 2023Okapi: Instruction-tuned Large Language Models in Multiple Languages with Reinforcement Learning from Human Feedbackcited by 190
  4. 2019Emerging Cross-lingual Structure in Pretrained Language Modelscited by 181
  5. 2023SIB-200: A Simple, Inclusive, and Big Evaluation Dataset for Topic Classification in 200+ Languages and Dialectscited by 152
  6. 2022Towards a Cleaner Document-Oriented Multilingual Crawled Corpuscited by 143
  7. 2023BUFFET: Benchmarking Large Language Models for Few-shot Cross-lingual Transfercited by 68
  8. 2023AfroBench: How Good are Large Language Models on African Languages?cited by 43
  9. 2024Towards Multilingual LLM Evaluation for European Languagescited by 30
  10. 2024Uhura: A Benchmark for Evaluating Scientific Question Answering and Truthfulness in Low-Resource African Languagescited by 8
  11. 2024GrEmLIn: A Repository of Green Baseline Embeddings for 87 Low-Resource Languages Injected with Multilingual Graph Knowledgecited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 7.

7 papers.

  1. 2026Orthogonal Model Mergingcited by 4
  2. 2026Abstract Activation Spaces for Content-Invariant Reasoning in Large Language Modelscited by 3
  3. 2026The GaoYao Benchmark: A Comprehensive Framework for Evaluating Multilingual and Multicultural Abilities of Large Language Modelscited by 1
  4. 2026MADE: Beyond Scoring via a Multilingual Agentic Diagnosing Engine for Fine-Grained Evaluation Insightscited by 1
  5. 2026Bootstrapping Embeddings for Low Resource Languagescited by 0
  6. 2026mmPISA-bench: Do LLMs Reason Equally Well Across 43 Languages?cited by 0
  7. 2026The Language-Energy Divide: Measuring Energy Costs of Multilingual LLM Inferencecited by 0