mlsanityData to 7 Oct 2026

A Principled Framework for Evaluating on Typologically Diverse Languages

arXiv 2407.05022 ↗First posted 6 Jul 2024cs.CLcited by 12

Not on the map. Why?

Abstract (arXiv)

Beyond individual languages, multilingual natural language processing (NLP) research increasingly aims to develop models that perform well across languages generally. However, evaluating these systems on all the world's languages is practically infeasible. To attain generalizability, representative language sampling is essential. Previous work argues that generalizable multilingual evaluation sets should contain languages with diverse typological properties. However, 'typologically diverse' language samples have been found to vary considerably in this regard, and popular sampling methods are flawed and inconsistent. We present a language sampling framework for selecting highly typologically diverse languages given a sampling frame, informed by language typology. We compare sampling methods with a range of metrics and find that our systematic methods consistently retrieve more typologically diverse language selections than previous methods in NLP. Moreover, we provide evidence that this affects generalizability in multilingual model evaluation, emphasizing the importance of diverse language sampling in NLP evaluation.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Builds on 22 of its 71 references that are in the map. First posted on arXiv 6 Jul 2024.

Builds on

22 of 71 references.

References · 22

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  3. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  4. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,462
  5. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  6. 2022No Language Left Behind: Scaling Human-Centered Machine Translationcited by 1,290
  7. 2020TyDi QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse Languagescited by 486
  8. 2024Multilingual E5 Text Embeddings: A Technical Reportcited by 357
  9. 2020XCOPA: A Multilingual Dataset for Causal Commonsense Reasoningcited by 330
  10. 2020How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Modelscited by 313
  11. 2022Challenges and Strategies in Cross-Cultural NLPcited by 216
  12. 2023Language Model Tokenizers Introduce Unfairness Between Languagescited by 207
10 more
  1. 2023Do All Languages Cost the Same? Tokenization in the Era of Commercial Language Modelscited by 186
  2. 2024Aya Dataset: An Open-Access Collection for Multilingual Instruction Tuningcited by 174
  3. 2023Glot500: Scaling Multilingual Corpora and Language Models to 500 Languagescited by 116
  4. 2023Don't Trust ChatGPT when Your Question is not in English: A Study of Multilingual Abilities and Types of LLMscited by 103
  5. 2020It's Easier to Translate out of English than into it: Measuring Neural Translation Difficulty by Cross-Mutual Informationcited by 26
  6. 2021Subword Pooling Makes a Differencecited by 18
  7. 2023SLABERT Talk Pretty One Day: Modeling Second Language Acquisition with BERTcited by 15
  8. 2020SIGTYP 2020 Shared Task: Prediction of Typological Featurescited by 13
  9. 2023Average Is Not Enough: Caveats of Multilingual Evaluationcited by 5
  10. 2024Measuring diversity. A review and an empirical analysiscited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 4; 2026: 7.

12 papers.

  1. 2025Modality Matching Matters: Calibrating Language Distances for Cross-Lingual Transfer in URIEL+cited by 3
  2. 2026Do LLMs Really Memorize Personally Identifiable Information? Revisiting PII Leakage with a Cue-Controlled Memorization Frameworkcited by 3
  3. 2025NLP Security and Ethics, in the Wildcited by 2
  4. 2025Beyond Weaponization: NLP Security for Medium and Lower-Resourced Languages in Their Own Rightcited by 2
  5. 2024Recipe for Zero-shot POS Tagging: Is It Useful in Realistic Scenarios?cited by 1
  6. 2025On the Interplay between Positional Encodings, Morphological Complexity, and Word Order Flexibilitycited by 1
  7. 2026Typologically Informed Parameter Aggregationcited by 1
  8. 2026Polyglot Teachers: Evaluating Language Models for Multilingual Synthetic Data Generationcited by 1
  9. 2026emb-diversity: A Tool for Embedding-Based Measurement of Data Diversitycited by 1
  10. 2026Vocabulary shapes cross-lingual variation of word-order learnability in language modelscited by 0
  11. 2026Pluralis v0.1: Towards a Multicultural, Multimodal, Multilingual Benchmark for AI Risk and Reliabilitycited by 0
  12. 2026CoSE-E: A Benchmark for Code-switched Speech Evaluation in Enterprise Settingscited by 0