mlsanityData to 14 Oct 2026

Systematic Inequalities in Language Technology Performance across the World's Languages

arXiv 2110.06733 ↗First posted 13 Oct 2021cs.CLcited by 120

Not on the map. Why?

Abstract (arXiv)

Natural language processing (NLP) systems have become a central technology in communication, education, medicine, artificial intelligence, and many other domains of research and development. While the performance of NLP methods has grown enormously over the last decade, this progress has been restricted to a minuscule subset of the world's 6,500 languages. We introduce a framework for estimating the global utility of language technologies as revealed in a comprehensive snapshot of recent publications in NLP. Our analyses involve the field at large, but also more in-depth studies on both user-facing technologies (machine translation, language understanding, question answering, text-to-speech synthesis) as well as more linguistic NLP tasks (dependency parsing, morphological inflection). In the process, we (1) quantify disparities in the current state of NLP research, (2) explore some of its associated societal and academic factors, and (3) produce tailored recommendations for evidence-based policy making aimed at promoting more global and equitable language technologies.

Key facts. Cited by 120 papers in the map (data through 14 Oct 2026). Builds on 27 of its 70 references that are in the map. First posted on arXiv 13 Oct 2021.

Builds on

27 of 70 references.

References · 27

  1. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  2. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,181
  3. 2018XNLI: Evaluating Cross-lingual Sentence Representationscited by 915
  4. 2016Semantics derived automatically from language corpora contain human-like biasescited by 901
  5. 2020The State and Fate of Linguistic Diversity and Inclusion in the NLP Worldcited by 798
  6. 2020FastSpeech 2: Fast and High-Quality End-to-End Text to Speechcited by 717
  7. 2020TyDi QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse Languagescited by 488
  8. 2019Bias in Bios: A Case Study of Semantic Representation Bias in a High-Stakes Settingcited by 349
  9. 2019MLQA: Evaluating Cross-lingual Extractive Question Answeringcited by 340
  10. 2018AI Fairness 360: An Extensible Toolkit for Detecting, Understanding, and Mitigating Unwanted Algorithmic Biascited by 319
  11. 2019Massively Multilingual Neural Machine Translation in the Wild: Findings and Challengescited by 257
  12. 2019Choosing Transfer Languages for Cross-Lingual Learningcited by 144
15 more
  1. 2020When Being Unseen from mBERT is just the Beginning: Handling New Languages With Multilingual Language Modelscited by 131
  2. 2018Construction of the Literature Graph in Semantic Scholarcited by 118
  3. 2015Practical Bayesian model evaluation using leave-one-out cross-validation and WAICcited by 110
  4. 2018A Unified Approach to Quantifying Algorithmic Unfairness: Measuring Individual & Group Unfairness via Inequality Indicescited by 97
  5. 201975 Languages, 1 Model: Parsing Universal Dependencies Universallycited by 90
  6. 2018Has Machine Translation Achieved Human Parity? A Case for Document-level Evaluationcited by 88
  7. 2021Including Signed Languages in Natural Language Processingcited by 68
  8. 2020Predicting Performance for Natural Language Processing Taskscited by 49
  9. 2019The SIGMORPHON 2019 Shared Task: Morphological Analysis in Context and Cross-Lingual Transfer for Inflectioncited by 43
  10. 2018The CoNLL--SIGMORPHON 2018 Shared Task: Universal Morphological Reinflectioncited by 41
  11. 2017CoNLL-SIGMORPHON 2017 Shared Task: Universal Morphological Reinflection in 52 Languagescited by 40
  12. 2020SIGMORPHON 2020 Shared Task 0: Typologically Diverse Morphological Inflectioncited by 39
  13. 2021SD-QA: Spoken Dialectal Question Answering for the Real Worldcited by 38
  14. 2020From SPMRL to NMRL: What Did We Learn (and Unlearn) in a Decade of Parsing Morphologically-Rich Languages (MRLs)?cited by 26
  15. 2014Predicting Crowd Behavior with Big Public Datacited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 1; 2022: 21; 2023: 29; 2024: 25; 2025: 27; 2026: 17.

120 papers.

Sort:

By area:

  1. 2023Low-Resource Languages Jailbreak GPT-4cited by 309
  2. 2023MEGA: Multilingual Evaluation of Generative AIcited by 293
  3. 2023UniMax: Fairer and more Effective Language Sampling for Large-Scale Multilingual Pretrainingcited by 151
  4. 2024Multi-IF: Benchmarking LLMs on Multi-Turn and Multilingual Instructions Followingcited by 102
  5. 2023NLPositionality: Characterizing Design Biases of Datasets and Modelscited by 94
  6. 2024The Language Barrier: Dissecting Safety Challenges of LLMs in Multilingual Contextscited by 94
  7. 2022GLUE-X: Evaluating Natural Language Understanding Models from an Out-of-distribution Generalization Perspectivecited by 72
  8. 2023BUFFET: Benchmarking Large Language Models for Few-shot Cross-lingual Transfercited by 68
  9. 2024Why do language models perform worse for morphologically complex languages?cited by 62
  10. 2022Some Languages are More Equal than Others: Probing Deeper into the Linguistic Disparity in the NLP Worldcited by 55
  11. 2022IGLUE: A Benchmark for Transfer Learning across Modalities, Tasks, and Languagescited by 54
  12. 2022Multi-VALUE: A Framework for Cross-Dialectal English NLPcited by 53
38 more
  1. 2025How Linguistics Learned to Stop Worrying and Love the Language Modelscited by 51
  2. 2022Evaluation Gaps in Machine Learning Practicecited by 49
  3. 2025The State of Multilingual LLM Safety Research: From Measuring the Language Gap to Mitigating Itcited by 45
  4. 2022Expanding Pretrained Models to Thousands More Languages via Lexicon-based Adaptationcited by 44
  5. 2024SEACrowd: A Multilingual Multimodal Data Hub and Benchmark Suite for Southeast Asian Languagescited by 43
  6. 2024Language Ranker: A Metric for Quantifying LLM Performance Across High and Low-Resource Languagescited by 38
  7. 2024Evaluating Cultural Adaptability of a Large Language Model via Simulation of Synthetic Personascited by 37
  8. 2023XTREME-UP: A User-Centric Scarce-Data Benchmark for Under-Represented Languagescited by 34
  9. 2022Data Governance in the Age of Large-Scale Data-Driven Language Technologycited by 33
  10. 2023Quantifying the Dialect Gap and its Correlates Across Languagescited by 33
  11. 2023When Is Multilinguality a Curse? Language Modeling for 250 High- and Low-Resource Languagescited by 32
  12. 2024SELF-GUIDE: Better Task-Specific Instruction Following via Self-Synthetic Finetuningcited by 32
  13. 2024mHumanEval -- A Multilingual Benchmark to Evaluate Large Language Models for Code Generationcited by 32
  14. 2023Dissociating language and thought in large language modelscited by 29
  15. 2022Bloom Library: Multimodal Datasets in 300+ Languages for a Variety of Downstream Taskscited by 28
  16. 2022Beyond Counting Datasets: A Survey of Multilingual Dataset Construction and Necessary Resourcescited by 25
  17. 2024Exploring Multilingual Concepts of Human Value in Large Language Models: Is Value Alignment Consistent, Transferable and Controllable across Languages?cited by 24
  18. 2021Dataset Geography: Mapping Language Data to Language Userscited by 22
  19. 2024AboutMe: Using Self-Descriptions in Webpages to Document the Effects of English Pretraining Data Filterscited by 22
  20. 2022Not always about you: Prioritizing community needs when developing endangered language technologycited by 21
  21. 2022Evaluating the Diversity, Equity and Inclusion of NLP Technology: A Case Study for Indian Languagescited by 20
  22. 2022How can NLP Help Revitalize Endangered Languages? A Case Study and Roadmap for the Cherokee Languagecited by 19
  23. 2024Cendol: Open Instruction-tuned Generative Large Language Models for Indonesian Languagescited by 18
  24. 2024No Filter: Cultural and Socioeconomic Diversity in Contrastive Vision-Language Modelscited by 16
  25. 2025Grounding Multilingual Multimodal LLMs With Cultural Knowledgecited by 16
  26. 2024What Drives Performance in Multilingual Language Models?cited by 15
  27. 2022MDIA: A Benchmark for Multilingual Dialogue Generation in 46 Languagescited by 13
  28. 2022MIA 2022 Shared Task: Evaluating Cross-lingual Open-Retrieval Question Answering for 16 Diverse Languagescited by 11
  29. 2023Bridging the Digital Divide: Performance Variation across Socio-Economic Factors in Vision-Language Modelscited by 11
  30. 2024Back to School: Translation Using Grammar Bookscited by 11
  31. 2023Language Embeddings Sometimes Contain Typological Generalizationscited by 9
  32. 2023GlobalBench: A Benchmark for Global Progress in Natural Language Processingcited by 9
  33. 2023BIG-C: a Multimodal Multi-Purpose Dataset for Bembacited by 9
  34. 2023CODET: A Benchmark for Contrastive Dialectal Evaluation of Machine Translationcited by 9
  35. 2023Translation-Enhanced Multilingual Text-to-Image Generationcited by 9
  36. 2023The timing bottleneck: Why timing and overlap are mission-critical for conversational user interfaces, speech recognition and dialogue systemscited by 9
  37. 2023Social Contract AI: Aligning AI Assistants with Implicit Group Normscited by 9
  38. 2022DataLab: A Platform for Data Analysis and Interventioncited by 8

50 most cited of 120 All 120 (JSON)