mlsanityData to 14 Oct 2026

XTREME-UP: A User-Centric Scarce-Data Benchmark for Under-Represented Languages

arXiv 2305.11938 ↗First posted 19 May 2023cs.CLcited by 34

Not on the map. Why?

Abstract (arXiv)

Data scarcity is a crucial issue for the development of highly multilingual NLP systems. Yet for many under-represented languages (ULs) -- languages for which NLP re-search is particularly far behind in meeting user needs -- it is feasible to annotate small amounts of data. Motivated by this, we propose XTREME-UP, a benchmark defined by: its focus on the scarce-data scenario rather than zero-shot; its focus on user-centric tasks -- tasks with broad adoption by speakers of high-resource languages; and its focus on under-represented languages where this scarce-data scenario tends to be most realistic. XTREME-UP evaluates the capabilities of language models across 88 under-represented languages over 9 key user-centric technologies including ASR, OCR, MT, and information access tasks that are of general utility. We create new datasets for OCR, autocomplete, semantic parsing, and transliteration, and build on and refine existing datasets for other tasks. XTREME-UP provides methodology for evaluating many modeling scenarios including text-only, multi-modal (vision, audio, and text),supervised parameter tuning, and in-context learning. We evaluate commonly used models on the benchmark. We release all code and scripts to train and evaluate models

Key facts. Cited by 34 papers in the map (data through 14 Oct 2026). Builds on 53 of its 103 references that are in the map. First posted on arXiv 19 May 2023.

Builds on

53 of 103 references.

References · 53

  1. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  2. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  3. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,818
  4. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,554
  5. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,392
  6. 2020The State and Fate of Linguistic Diversity and Inclusion in the NLP Worldcited by 798
  7. 2020XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalizationcited by 742
  8. 2021The FLORES-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translationcited by 560
  9. 2022Efficiently Scaling Transformer Inferencecited by 539
  10. 2021ByT5: Towards a token-free future with pre-trained byte-to-byte modelscited by 509
  11. 2020TyDi QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse Languagescited by 488
  12. 2022FLEURS: Few-shot Learning Evaluation of Universal Representations of Speechcited by 469
38 more
  1. 2021XL-Sum: Large-Scale Multilingual Abstractive Summarization for 44 Languagescited by 347
  2. 2023MEGA: Multilingual Evaluation of Generative AIcited by 293
  3. 2018A Survey on Deep Learning for Named Entity Recognitioncited by 282
  4. 2020Universal Dependencies v2: An Evergrowing Multilingual Treebank Collectioncited by 271
  5. 2020XGLUE: A New Benchmark Dataset for Cross-lingual Pre-training, Understanding and Generationcited by 222
  6. 2022Scaling Up Models and Data with textttt5x and textttseqiocited by 187
  7. 2021XTREME-R: Towards More Challenging and Nuanced Multilingual Evaluationcited by 168
  8. 2020Participatory Research for Low-resourced Machine Translation: A Case Study in African Languagescited by 159
  9. 2020MTOP: A Comprehensive Multilingual Task-Oriented Semantic Parsing Benchmarkcited by 158
  10. 2021MasakhaNER: Named Entity Recognition for African Languagescited by 155
  11. 2020XOR QA: Cross-lingual Open-Retrieval Question Answeringcited by 121
  12. 2021Systematic Inequalities in Language Technology Performance across the World's Languagescited by 120
  13. 2022Compositional Semantic Parsing with Large Language Modelscited by 95
  14. 2021Survey of Low-Resource Machine Translationcited by 92
  15. 2018NVIDIA Tensor Core Programmability, Performance & Precisioncited by 79
  16. 2020Language ID in the Wild: Unexpected Challenges on the Path to a Thousand-Language Web Text Corpuscited by 77
  17. 2022Building Machine Translation Systems for the Next Thousand Languagescited by 69
  18. 2022NusaX: Multilingual Parallel Sentiment Dataset for 10 Indonesian Local Languagescited by 68
  19. 2023AfriSenti: A Twitter Sentiment Analysis Benchmark for African Languagescited by 68
  20. 2022NaijaSenti: A Nigerian Twitter Sentiment Corpus for Multilingual Sentiment Analysiscited by 64
  21. 2021BiToD: A Bilingual Multi-Domain Dataset For Task-Oriented Dialogue Modelingcited by 56
  22. 2020Transfer Learning and Distant Supervision for Multilingual Transformer Models: A Study on African Languagescited by 52
  23. 2020A Closer Look at Few-Shot Crosslingual Transfer: The Choice of Shots Matterscited by 45
  24. 2022MasakhaNER 2.0: Africa-centric Transfer Learning for Named Entity Recognitioncited by 38
  25. 2020A Call for More Rigor in Unsupervised Cross-lingual Learningcited by 37
  26. 2018Dissecting the NVIDIA Volta GPU Architecture via Microbenchmarkingcited by 35
  27. 2020Processing South Asian Languages Written in the Latin Script: the Dakshina Datasetcited by 35
  28. 2020Is Supervised Syntactic Parsing Beneficial for Language Understanding? An Empirical Investigationcited by 34
  29. 2020KINNEWS and KIRNEWS: Benchmarking Cross-Lingual Text Classification for Kinyarwanda and Kirundicited by 31
  30. 2020OCR Post Correction for Endangered Language Textscited by 31
  31. 2021GlobalWoZ: Globalizing MultiWoZ to Develop Multilingual Task-Oriented Dialogue Systemscited by 31
  32. 2022Cross-Lingual Dialogue Dataset Creation via Outline-Based Generationcited by 23
  33. 2023PRESTO: A Multilingual Dataset for Parsing Realistic Task-Oriented Dialogscited by 16
  34. 2022MULTI3NLU++: A Multilingual, Multi-Intent, Multi-Domain Dataset for Natural Language Understanding in Task-Oriented Dialoguecited by 15
  35. 2019Scale MLPerf-0.6 models on Google TPU-v3 Podscited by 14
  36. 2022Maestro-U: Leveraging joint speech-text representation learning for zero supervised speech ASRcited by 13
  37. 2021Lexically Aware Semi-Supervised Learning for OCR Post-Correctioncited by 12
  38. 2022OCR Improves Machine Translation for Low-Resource Languagescited by 5

50 most cited of 53 All 53 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 12; 2024: 12; 2025: 8; 2026: 2.

34 papers.

Sort:

By area:

  1. 2024Aya Dataset: An Open-Access Collection for Multilingual Instruction Tuningcited by 177
  2. 2025EmbeddingGemma: Powerful and Lightweight Text Representationscited by 139
  3. 2025Gemini Embedding: Generalizable Embeddings from Geminicited by 94
  4. 2023BUFFET: Benchmarking Large Language Models for Few-shot Cross-lingual Transfercited by 68
  5. 2024IndicGenBench: A Multilingual Benchmark to Evaluate Generation Capabilities of LLMs on Indic Languagescited by 56
  6. 2024DIALECTBENCH: A NLP Benchmark for Dialects, Varieties, and Closely-Related Languagescited by 50
  7. 2023AfroBench: How Good are Large Language Models on African Languages?cited by 43
  8. 2024SEACrowd: A Multilingual Multimodal Data Hub and Benchmark Suite for Southeast Asian Languagescited by 43
  9. 2024MYTE: Morphology-Driven Byte Encoding for Better and Fairer Multilingual Language Modelingcited by 40
  10. 2023Quantifying the Dialect Gap and its Correlates Across Languagescited by 33
  11. 2023SiRA: Sparse Mixture of Low Rank Adaptationcited by 32
  12. 2023Revisiting non-English Text Simplification: A Unified Multilingual Benchmarkcited by 27
22 more
  1. 2024Evaluating the Elementary Multilingual Capabilities of Large Language Models with MultiQcited by 27
  2. 2023Leveraging LLMs for Synthesizing Training Data Across Many Languages in Multilingual Dense Retrievalcited by 24
  3. 2025Charting the Landscape of African NLP: Mapping Progress and Shaping the Road Aheadcited by 16
  4. 2024CoBa: Convergence Balancer for Multitask Finetuning of Large Language Modelscited by 13
  5. 2024Analyzing and Adapting Large Language Models for Few-Shot Multilingual NLU: Are We There Yet?cited by 11
  6. 2024Translation Errors Significantly Impact Low-Resource Languages in Cross-Lingual Learningcited by 9
  7. 2026Massive Sound Embedding Benchmark (MSEB)cited by 9
  8. 2026Learning Retrieval Models with Sparse Autoencoderscited by 6
  9. 2024On the Evaluation Practices in Multilingual NLP: Can Machine Translation Offer an Alternative to Human Translations?cited by 5
  10. 2025MiLiC-Eval: Benchmarking Multilingual LLMs for China's Minority Languagescited by 5
  11. 2023A Systematic Study of Performance Disparities in Multilingual Task-Oriented Dialogue Systemscited by 4
  12. 2023Improving Cross-Lingual Transfer through Subtree-Aware Word Reorderingcited by 4
  13. 2024Fine-tuning Large Language Models with Limited Data: A Survey and Practical Guidecited by 4
  14. 2025Trans-EnV: A Framework for Evaluating the Linguistic Robustness of LLMs Against English Varietiescited by 4
  15. 2023SQATIN: Supervised Instruction Tuning Meets Question Answering for Improved Dialogue NLUcited by 3
  16. 2023Optimal Transport Posterior Alignment for Cross-lingual Semantic Parsingcited by 2
  17. 2023FIAT: Fusing learning paradigms with Instruction-Accelerated Tuningcited by 2
  18. 2025Bridging Gaps in Natural Language Processing for Yor`ub'a: A Systematic Review of a Decade of Progress and Prospectscited by 2
  19. 2025Revisiting Projection-based Data Transfer for Cross-Lingual Named Entity Recognition in Low-Resource Languagescited by 1
  20. 2023CUNI Submission to MRL 2023 Shared Task on Multi-lingual Multi-task Information Retrievalcited by 0
  21. 2024MaiNLP at SemEval-2024 Task 1: Analyzing Source Language Selection in Cross-Lingual Textual Relatednesscited by 0
  22. 2025Opportunities and Challenges of Natural Language Processing for Low-Resource Senegalese Languages in Social Science Researchcited by 0