mlsanityData to 14 Oct 2026

MEXA: Multilingual Evaluation of English-Centric LLMs via Cross-Lingual Alignment

arXiv 2410.05873 ↗First posted 8 Oct 2024cs.CL cs.AIcited by 25

Not on the map. Why?

Abstract (arXiv)

English-centric large language models (LLMs) often show strong multilingual capabilities. However, their multilingual performance remains unclear and is under-evaluated for many other languages. Most benchmarks for multilinguality focus on classic NLP tasks or cover a minimal number of languages. We introduce MEXA, a method for assessing the multilingual capabilities of pre-trained English-centric LLMs using parallel sentences, which are available for more languages than existing downstream tasks. MEXA leverages that English-centric LLMs use English as a pivot language in their intermediate layers. MEXA computes the alignment between English and non-English languages using parallel sentences to evaluate the transfer of language understanding from English to other languages. This alignment can be used to estimate model performance in different languages. We conduct controlled experiments using various parallel datasets (FLORES-200 and Bible), models (Llama family, Gemma family, Mistral, and OLMo), and established downstream tasks (Belebele, m-MMLU, and m-ARC). We explore different methods to compute embeddings in decoder-only models. Our results show that MEXA, in its default settings, achieves an average Pearson correlation of 0.90 between its predicted scores and actual task performance across languages. This suggests that MEXA is a reliable method for estimating the multilingual capabilities of English-centric LLMs, providing a clearer understanding of their multilingual potential and the inner workings of LLMs. Leaderboard: https://cis-lmu-mexa.hf.space, Code: https://github.com/cisnlp/MEXA.

Key facts. Cited by 25 papers in the map (data through 14 Oct 2026). Builds on 78 of its 111 references that are in the map. First posted on arXiv 8 Oct 2024.

Builds on

78 of 111 references.

References · 78

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2023GPT-4 Technical Reportcited by 18,396
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  6. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  7. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  8. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  9. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
  10. 2023Mistral 7Bcited by 3,164
  11. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,984
  12. 2024Mixtral of Expertscited by 1,823
38 more
  1. 2019Similarity of Neural Network Representations Revisitedcited by 1,755
  2. 2016Enriching Word Vectors with Subword Informationcited by 1,583
  3. 2019Cross-lingual Language Model Pretrainingcited by 1,090
  4. 2018XNLI: Evaluating Cross-lingual Sentence Representationscited by 915
  5. 2020The State and Fate of Linguistic Diversity and Inclusion in the NLP Worldcited by 798
  6. 2019How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddingscited by 784
  7. 2020XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalizationcited by 742
  8. 2024OLMo: Accelerating the Science of Language Modelscited by 613
  9. 2020Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillationcited by 593
  10. 2019On the Cross-lingual Transferability of Monolingual Representationscited by 578
  11. 2020TyDi QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse Languagescited by 488
  12. 2021XL-Sum: Large-Scale Multilingual Abstractive Summarization for 44 Languagescited by 347
  13. 2019MLQA: Evaluating Cross-lingual Extractive Question Answeringcited by 340
  14. 2020XCOPA: A Multilingual Dataset for Causal Commonsense Reasoningcited by 333
  15. 2023The Belebele Benchmark: a Parallel Reading Comprehension Dataset in 122 Language Variantscited by 311
  16. 2023Improving Text Embeddings with Large Language Modelscited by 303
  17. 2023MEGA: Multilingual Evaluation of Generative AIcited by 293
  18. 2024Do Llamas Work in English? On the Latent Language of Multilingual Transformerscited by 281
  19. 2021Few-shot Learning with Multilingual Language Modelscited by 278
  20. 2020Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translationcited by 277
  21. 2023Fine-Tuning LLaMA for Multi-Stage Text Retrievalcited by 272
  22. 2019PAWS-X: A Cross-lingual Adversarial Dataset for Paraphrase Identificationcited by 264
  23. 2023ChatGPT Beyond English: Towards a Comprehensive Evaluation of Large Language Models in Multilingual Learningcited by 206
  24. 2022SGPT: GPT Sentence Embeddings for Semantic Searchcited by 204
  25. 2023Okapi: Instruction-tuned Large Language Models in Multiple Languages with Reinforcement Learning from Human Feedbackcited by 190
  26. 2019Emerging Cross-lingual Structure in Pretrained Language Modelscited by 181
  27. 2020WikiLingua: A New Benchmark Dataset for Cross-Lingual Abstractive Summarizationcited by 154
  28. 2023Having Beer after Prayer? Measuring Cultural Bias in Large Language Modelscited by 153
  29. 2023IndicTrans2: Towards High-Quality and Accessible Machine Translation Models for all 22 Scheduled Indian Languagescited by 152
  30. 2024How do Large Language Models Handle Multilingualism?cited by 150
  31. 2023What's In My Big Data?cited by 129
  32. 2020Multilingual Alignment of Contextual Word Representationscited by 114
  33. 2023M3Exam: A Multilingual, Multimodal, Multilevel Benchmark for Examining Large Language Modelscited by 110
  34. 2022Towards Leaving No Indic Language Behind: Building Monolingual Corpora, Benchmark and Models for Indic Languagescited by 102
  35. 2019Unicoder: A Universal Language Encoder by Pre-training with Multiple Cross-lingual Taskscited by 96
  36. 2024KMMLU: Measuring Massive Multitask Language Understanding in Koreancited by 91
  37. 2024ArabicMMLU: Assessing Massive Multitask Language Understanding in Arabiccited by 88
  38. 2023Scaling Sentence Embeddings with Large Language Modelscited by 85

50 most cited of 78 All 78 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 12; 2026: 12.

25 papers.

Sort:

By area:

  1. 2025A Survey of Multilingual Reasoning in Language Modelscited by 33
  2. 2025Middle-Layer Representation Alignment for Cross-Lingual Transfer in Fine-Tuned LLMscited by 29
  3. 2025Multilingual Routing in Mixture-of-Expertscited by 29
  4. 2024GlotCC: An Open Broad-Coverage CommonCrawl Corpus and Pipeline for Minority Languagescited by 16
  5. 2025Investigating and Scaling up Code-Switching for Multilingual Language Model Pre-Trainingcited by 16
  6. 2025Can you map it to English? The Role of Cross-Lingual Alignment in Multilingual Performance of LLMscited by 12
  7. 2025From Neurons to Semantics: Evaluating Cross-Linguistic Alignment Capabilities of Large Language Models via Neurons Alignmentcited by 9
  8. 2025How Programming Concepts and Neurons Are Shared in Code Language Modelscited by 5
  9. 2025On the Entity-Level Alignment in Crosslingual Consistencycited by 4
  10. 2025Text2Cypher Across Languages: Evaluating and Finetuning LLMscited by 3
  11. 2026Can Embedding Similarity Predict Cross-Lingual Transfer? A Systematic Study on African Languagescited by 3
  12. 2025ToxSyn-PT: A Synthetic Fine-Grained Dataset of Minority-Targeted Toxic Language in Portuguesecited by 2
13 more
  1. 2026Efficient Multilingual Reasoning Transfer via Progressive Code-Switchingcited by 2
  2. 2025The Effect of Language Diversity When Fine-Tuning Large Language Models for Translationcited by 1
  3. 2026Quantifying Cross-Lingual Transfer in Paralinguistic Speech Taskscited by 1
  4. 2025ALOPE: Adaptive Layer Optimization for Translation Quality Estimation using Large Language Modelscited by 0
  5. 2026Incremental Multilingual Text2Cypher with Adapter Combinationcited by 0
  6. 2026Neither Here Nor There: Cross-Lingual Representation Dynamics of Code-Mixed Text in Multilingual Encoderscited by 0
  7. 2026Language Shapes Instruction Hierarchy Compliance in Multilingual LLMscited by 0
  8. 2026Bridging the English-Arabic Medical Knowledge Gap: Targeted Low-Rank Adaptation via Causal Layer Selectioncited by 0
  9. 2026Predicting Multilingual Classification and Translation Performance of LLMs with Cross-Lingual Alignment -- Is English Enough?cited by 0
  10. 2026A Systematic Comparison of Multilingual Interpretability Methods Reveals Anisotropy-Driven Failurescited by 0
  11. 2026Cross-Lingual Representation Alignment by Token-Level Optimal Transport in a Language-Agnostic Spacecited by 0
  12. 2026Online Language Adaptive Sampling for Better Distributed Cross-lingual Gainscited by 0
  13. 2026Feeding BabyLMs Macaroni: Code-Switching Curricula Cause Cross-Lingual Convergencecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025AlignX: Advancing Multilingual Large Language Models with Multilingual Representation Alignment
  2. 2024Getting More from Less: Large Language Models are Good Spontaneous Multilingual Learners
  3. 2025MMLU-ProX: A Multilingual Benchmark for Advanced Large Language Model Evaluation
  4. 2025Aligning LLMs for Multilingual Consistency in Enterprise Applications
  5. 2024Concept Space Alignment in Multilingual LLMs
  6. 2023Extrapolating Large Language Models to Non-English by Aligning Languages
  7. 2025From Unaligned to Aligned: Scaling Multilingual LLMs with Multi-Way Parallel Corpora
  8. 2024A Survey on Multilingual Large Language Models: Corpora, Alignment, and Bias