# A Principled Framework for Evaluating on Typologically Diverse Languages

arXiv: https://arxiv.org/abs/2407.05022 · HTML: https://mlsanity.com/paper/2407.05022/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 71 references that are in the map.
- First posted on arXiv 6 Jul 2024.

## Builds on (most cited first, 22 of 22)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [No Language Left Behind: Scaling Human-Centered Machine Translation](https://mlsanity.com/paper/2207.04672/) (2022)
- [TyDi QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse Languages](https://mlsanity.com/paper/2003.05002/) (2020)
- [Multilingual E5 Text Embeddings: A Technical Report](https://mlsanity.com/paper/2402.05672/) (2024)
- [XCOPA: A Multilingual Dataset for Causal Commonsense Reasoning](https://mlsanity.com/paper/2005.00333/) (2020)
- [How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Models](https://mlsanity.com/paper/2012.15613/) (2020)
- [Challenges and Strategies in Cross-Cultural NLP](https://mlsanity.com/paper/2203.10020/) (2022)
- [Language Model Tokenizers Introduce Unfairness Between Languages](https://mlsanity.com/paper/2305.15425/) (2023)
- [Do All Languages Cost the Same? Tokenization in the Era of Commercial Language Models](https://mlsanity.com/paper/2305.13707/) (2023)
- [Aya Dataset: An Open-Access Collection for Multilingual Instruction Tuning](https://mlsanity.com/paper/2402.06619/) (2024)
- [Glot500: Scaling Multilingual Corpora and Language Models to 500 Languages](https://mlsanity.com/paper/2305.12182/) (2023)
- [Don't Trust ChatGPT when Your Question is not in English: A Study of Multilingual Abilities and Types of LLMs](https://mlsanity.com/paper/2305.16339/) (2023)
- [It's Easier to Translate out of English than into it: Measuring Neural Translation Difficulty by Cross-Mutual Information](https://mlsanity.com/paper/2005.02354/) (2020)
- [Subword Pooling Makes a Difference](https://mlsanity.com/paper/2102.10864/) (2021)
- [SLABERT Talk Pretty One Day: Modeling Second Language Acquisition with BERT](https://mlsanity.com/paper/2305.19589/) (2023)
- [SIGTYP 2020 Shared Task: Prediction of Typological Features](https://mlsanity.com/paper/2010.08246/) (2020)
- [Average Is Not Enough: Caveats of Multilingual Evaluation](https://mlsanity.com/paper/2301.01269/) (2023)
- [Measuring diversity. A review and an empirical analysis](https://mlsanity.com/paper/2401.12365/) (2024)

## Built on by (most cited first, 12 of 12)

- [Modality Matching Matters: Calibrating Language Distances for Cross-Lingual Transfer in URIEL+](https://mlsanity.com/paper/2510.19217/) (2025)
- [Do LLMs Really Memorize Personally Identifiable Information? Revisiting PII Leakage with a Cue-Controlled Memorization Framework](https://mlsanity.com/paper/2601.03791/) (2026)
- [NLP Security and Ethics, in the Wild](https://mlsanity.com/paper/2504.06669/) (2025)
- [Beyond Weaponization: NLP Security for Medium and Lower-Resourced Languages in Their Own Right](https://mlsanity.com/paper/2507.03473/) (2025)
- [Recipe for Zero-shot POS Tagging: Is It Useful in Realistic Scenarios?](https://mlsanity.com/paper/2410.10576/) (2024)
- [On the Interplay between Positional Encodings, Morphological Complexity, and Word Order Flexibility](https://mlsanity.com/paper/2511.08139/) (2025)
- [Typologically Informed Parameter Aggregation](https://mlsanity.com/paper/2601.16629/) (2026)
- [Polyglot Teachers: Evaluating Language Models for Multilingual Synthetic Data Generation](https://mlsanity.com/paper/2604.11290/) (2026)
- [emb-diversity: A Tool for Embedding-Based Measurement of Data Diversity](https://mlsanity.com/paper/2607.19848/) (2026)
- [Vocabulary shapes cross-lingual variation of word-order learnability in language models](https://mlsanity.com/paper/2603.19427/) (2026)
- [Pluralis v0.1: Towards a Multicultural, Multimodal, Multilingual Benchmark for AI Risk and Reliability](https://mlsanity.com/paper/2607.06196/) (2026)
- [CoSE-E: A Benchmark for Code-switched Speech Evaluation in Enterprise Settings](https://mlsanity.com/paper/2609.35645/) (2026)

## Abstract (arXiv)

Beyond individual languages, multilingual natural language processing (NLP) research increasingly aims to develop models that perform well across languages generally. However, evaluating these systems on all the world's languages is practically infeasible. To attain generalizability, representative language sampling is essential. Previous work argues that generalizable multilingual evaluation sets should contain languages with diverse typological properties. However, 'typologically diverse' language samples have been found to vary considerably in this regard, and popular sampling methods are flawed and inconsistent. We present a language sampling framework for selecting highly typologically diverse languages given a sampling frame, informed by language typology. We compare sampling methods with a range of metrics and find that our systematic methods consistently retrieve more typologically diverse language selections than previous methods in NLP. Moreover, we provide evidence that this affects generalizability in multilingual model evaluation, emphasizing the importance of diverse language sampling in NLP evaluation.