# The Roles of English in Evaluating Multilingual Language Models

arXiv: https://arxiv.org/abs/2412.08392 · HTML: https://mlsanity.com/paper/2412.08392/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 26 references that are in the map.
- First posted on arXiv 11 Dec 2024.

## Builds on (most cited first, 17 of 17)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [The State and Fate of Linguistic Diversity and Inclusion in the NLP World](https://mlsanity.com/paper/2004.09095/) (2020)
- [OpenAssistant Conversations -- Democratizing Large Language Model Alignment](https://mlsanity.com/paper/2304.07327/) (2023)
- [Language Models are Multilingual Chain-of-Thought Reasoners](https://mlsanity.com/paper/2210.03057/) (2022)
- [How Good Are GPT Models at Machine Translation? A Comprehensive Evaluation](https://mlsanity.com/paper/2302.09210/) (2023)
- [Few-shot Learning with Multilingual Language Models](https://mlsanity.com/paper/2112.10668/) (2021)
- [Lessons from the Trenches on Reproducible Evaluation of Language Models](https://mlsanity.com/paper/2405.14782/) (2024)
- [Aya Dataset: An Open-Access Collection for Multilingual Instruction Tuning](https://mlsanity.com/paper/2402.06619/) (2024)
- [SIB-200: A Simple, Inclusive, and Big Evaluation Dataset for Topic Classification in 200+ Languages and Dialects](https://mlsanity.com/paper/2309.07445/) (2023)
- [Don't Trust ChatGPT when Your Question is not in English: A Study of Multilingual Abilities and Types of LLMs](https://mlsanity.com/paper/2305.16339/) (2023)
- [Do Multilingual Language Models Think Better in English?](https://mlsanity.com/paper/2308.01223/) (2023)
- [BUFFET: Benchmarking Large Language Models for Few-shot Cross-lingual Transfer](https://mlsanity.com/paper/2305.14857/) (2023)
- [Towards a Common Understanding of Contributing Factors for Cross-Lingual Transfer in Multilingual Language Models: A Review](https://mlsanity.com/paper/2305.16768/) (2023)
- [A Call for More Rigor in Unsupervised Cross-lingual Learning](https://mlsanity.com/paper/2004.14958/) (2020)
- [Square One Bias in NLP: Towards a Multi-Dimensional Exploration of the Research Manifold](https://mlsanity.com/paper/2206.09755/) (2022)
- [Zero-shot Cross-lingual Transfer of Prompt-based Tuning with a Unified Multilingual Prompt](https://mlsanity.com/paper/2202.11451/) (2022)
- [What is "Typological Diversity" in NLP?](https://mlsanity.com/paper/2402.04222/) (2024)

## Built on by (most cited first, 7 of 7)

- [Can We Trust AI Benchmarks? An Interdisciplinary Review of Current Issues in AI Evaluation](https://mlsanity.com/paper/2502.06559/) (2025)
- [Blessing of Multilinguality: A Systematic Analysis of Multilingual In-Context Learning](https://mlsanity.com/paper/2502.11364/) (2025)
- [Form and Meaning in Intrinsic Multilingual Evaluations](https://mlsanity.com/paper/2601.10580/) (2026)
- [A Diagnostic Benchmark for Sweden-Related Factual Knowledge](https://mlsanity.com/paper/2510.21360/) (2025)
- [Parallel Universes, Parallel Languages: A Comprehensive Study on LLM-based Multilingual Counterfactual Example Generation](https://mlsanity.com/paper/2601.00263/) (2026)
- [GlotEval: A Test Suite for Massively Multilingual Evaluation of Large Language Models](https://mlsanity.com/paper/2504.04155/) (2025)
- [Assessing the Impact of Language Disparity on Multilingual Linguistic Ability in Large Language Models](https://mlsanity.com/paper/2610.00540/) (2026)

## Abstract (arXiv)

Multilingual natural language processing is getting increased attention, with numerous models, benchmarks, and methods being released for many languages. English is often used in multilingual evaluation to prompt language models (LMs), mainly to overcome the lack of instruction tuning data in other languages. In this position paper, we lay out two roles of English in multilingual LM evaluations: as an interface and as a natural language. We argue that these roles have different goals: task performance versus language understanding. This discrepancy is highlighted with examples from datasets and evaluation setups. Numerous works explicitly use English as an interface to boost task performance. We recommend to move away from this imprecise method and instead focus on furthering language understanding.