# SIB-200: A Simple, Inclusive, and Big Evaluation Dataset for Topic Classification in 200+ Languages and Dialects

arXiv: https://arxiv.org/abs/2309.07445 · HTML: https://mlsanity.com/paper/2309.07445/

## Key facts

- Cited by 152 papers in the map (data through 14 Oct 2026).
- Builds on 49 of its 66 references that are in the map.
- First posted on arXiv 14 Sep 2023.

## Builds on (most cited first, 30 of 49)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [A Survey of Large Language Models](https://mlsanity.com/paper/2303.18223/) (2023)
- [mT5: A massively multilingual pre-trained text-to-text transformer](https://mlsanity.com/paper/2010.11934/) (2020)
- [Multitask Prompted Training Enables Zero-Shot Task Generalization](https://mlsanity.com/paper/2110.08207/) (2021)
- [No Language Left Behind: Scaling Human-Centered Machine Translation](https://mlsanity.com/paper/2207.04672/) (2022)
- [DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharing](https://mlsanity.com/paper/2111.09543/) (2021)
- [Cross-lingual Language Model Pretraining](https://mlsanity.com/paper/1901.07291/) (2019)
- [Multilingual Denoising Pre-training for Neural Machine Translation](https://mlsanity.com/paper/2001.08210/) (2020)
- [XNLI: Evaluating Cross-lingual Sentence Representations](https://mlsanity.com/paper/1809.05053/) (2018)
- [The State and Fate of Linguistic Diversity and Inclusion in the NLP World](https://mlsanity.com/paper/2004.09095/) (2020)
- [XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalization](https://mlsanity.com/paper/2003.11080/) (2020)
- [The FLORES-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translation](https://mlsanity.com/paper/2106.03193/) (2021)
- [FLEURS: Few-shot Learning Evaluation of Universal Representations of Speech](https://mlsanity.com/paper/2205.12446/) (2022)
- [MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfer](https://mlsanity.com/paper/2005.00052/) (2020)
- [Harnessing the Power of LLMs in Practice: A Survey on ChatGPT and Beyond](https://mlsanity.com/paper/2304.13712/) (2023)
- [Scaling Speech Technology to 1,000+ Languages](https://mlsanity.com/paper/2305.13516/) (2023)
- [MLQA: Evaluating Cross-lingual Extractive Question Answering](https://mlsanity.com/paper/1910.07475/) (2019)
- [XCOPA: A Multilingual Dataset for Causal Commonsense Reasoning](https://mlsanity.com/paper/2005.00333/) (2020)
- [The Belebele Benchmark: a Parallel Reading Comprehension Dataset in 122 Language Variants](https://mlsanity.com/paper/2308.16884/) (2023)
- [Multilingual Translation with Extensible Multilingual Pretraining and Finetuning](https://mlsanity.com/paper/2008.00401/) (2020)
- [Few-shot Learning with Multilingual Language Models](https://mlsanity.com/paper/2112.10668/) (2021)
- [Universal Dependencies v2: An Evergrowing Multilingual Treebank Collection](https://mlsanity.com/paper/2004.10643/) (2020)
- [InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Training](https://mlsanity.com/paper/2007.07834/) (2020)
- [MuRIL: Multilingual Representations for Indian Languages](https://mlsanity.com/paper/2103.10730/) (2021)
- [XTREME-R: Towards More Challenging and Nuanced Multilingual Evaluation](https://mlsanity.com/paper/2104.07412/) (2021)
- [Adapting Pre-trained Language Models to African Languages via Multilingual Adaptive Fine-Tuning](https://mlsanity.com/paper/2204.06487/) (2022)
- [Lifting the Curse of Multilinguality by Pre-training Modular Transformers](https://mlsanity.com/paper/2205.06266/) (2022)
- [Glot500: Scaling Multilingual Corpora and Language Models to 500 Languages](https://mlsanity.com/paper/2305.12182/) (2023)

## Built on by (most cited first, 30 of 152)

- [Why do language models perform worse for morphologically complex languages?](https://mlsanity.com/paper/2411.14198/) (2024)
- [DIALECTBENCH: A NLP Benchmark for Dialects, Varieties, and Closely-Related Languages](https://mlsanity.com/paper/2403.11009/) (2024)
- [AfroBench: How Good are Large Language Models on African Languages?](https://mlsanity.com/paper/2311.07978/) (2023)
- [IrokoBench: A New Benchmark for African Languages in the Age of Large Language Models](https://mlsanity.com/paper/2406.03368/) (2024)
- [SEACrowd: A Multilingual Multimodal Data Hub and Benchmark Suite for Southeast Asian Languages](https://mlsanity.com/paper/2406.10118/) (2024)
- [When AI Benchmarks Plateau: A Systematic Study of Benchmark Saturation](https://mlsanity.com/paper/2602.16763/) (2026)
- [MAGNET: Improving the Multilingual Fairness of Language Models with Adaptive Gradient-Based Tokenization](https://mlsanity.com/paper/2407.08818/) (2024)
- [MaLA-500: Massive Language Adaptation of Large Language Models](https://mlsanity.com/paper/2401.13303/) (2024)
- [Evaluating Morphological Alignment of Tokenizers in 70 Languages](https://mlsanity.com/paper/2507.06378/) (2025)
- [Unlocking the Potential of Model Merging for Low-Resource Languages](https://mlsanity.com/paper/2407.03994/) (2024)
- [InkubaLM: A small language model for low-resource African languages](https://mlsanity.com/paper/2408.17024/) (2024)
- [Evaluating the Capabilities of Large Language Models for Multi-label Emotion Understanding](https://mlsanity.com/paper/2412.17837/) (2024)
- [EMMA-500: Enhancing Massively Multilingual Adaptation of Large Language Models](https://mlsanity.com/paper/2409.17892/) (2024)
- [Charting the Landscape of African NLP: Mapping Progress and Shaping the Road Ahead](https://mlsanity.com/paper/2505.21315/) (2025)
- [What Drives Performance in Multilingual Language Models?](https://mlsanity.com/paper/2404.19159/) (2024)
- [Breaking the Script Barrier in Multilingual Pre-Trained Language Models with Transliteration-Based Post-Training Alignment](https://mlsanity.com/paper/2406.19759/) (2024)
- [Better to Ask in English: Evaluation of Large Language Models on English, Low-resource and Cross-Lingual Settings](https://mlsanity.com/paper/2410.13153/) (2024)
- [Multilingual Pretraining for Pixel Language Models](https://mlsanity.com/paper/2505.21265/) (2025)
- [AfriqueLLM: How Data Mixing and Model Architecture Impact Continued Pre-training for African Languages](https://mlsanity.com/paper/2601.06395/) (2026)
- [F2LLM-v2: Inclusive, Performant, and Efficient Embeddings for a Multilingual World](https://mlsanity.com/paper/2603.19223/) (2026)
- [LexC-Gen: Generating Data for Extremely Low-Resource Languages with Large Language Models and Bilingual Lexicons](https://mlsanity.com/paper/2402.14086/) (2024)
- [SambaLingo: Teaching Large Language Models New Languages](https://mlsanity.com/paper/2404.05829/) (2024)
- [MINERS: Multilingual Language Models as Semantic Retrievers](https://mlsanity.com/paper/2406.07424/) (2024)
- [The Responsible Foundation Model Development Cheatsheet: A Review of Tools & Resources](https://mlsanity.com/paper/2406.16746/) (2024)
- [AFRIDOC-MT: Document-level MT Corpus for African Languages](https://mlsanity.com/paper/2501.06374/) (2025)
- [AfriHate: A Multilingual Collection of Hate Speech and Abusive Language Datasets for African Languages](https://mlsanity.com/paper/2501.08284/) (2025)
- [Analyzing the Effect of Linguistic Similarity on Cross-Lingual Transfer: Tasks and Experimental Setups Matter](https://mlsanity.com/paper/2501.14491/) (2025)
- [Where Are We? Evaluating LLM Performance on African Languages](https://mlsanity.com/paper/2502.19582/) (2025)
- [Rethinking Multilingual Continual Pretraining: Data Mixing for Adapting LLMs Across Languages and Resources](https://mlsanity.com/paper/2504.04152/) (2025)
- [SSA-COMET: Do LLMs Outperform Learned Metrics in Evaluating MT for Under-Resourced African Languages?](https://mlsanity.com/paper/2506.04557/) (2025)

## Abstract (arXiv)

Despite the progress we have recorded in the last few years in multilingual natural language processing, evaluation is typically limited to a small set of languages with available datasets which excludes a large number of low-resource languages. In this paper, we created SIB-200 -- a large-scale open-sourced benchmark dataset for topic classification in 200 languages and dialects to address the lack of evaluation dataset for Natural Language Understanding (NLU). For many of the languages covered in SIB-200, this is the first publicly available evaluation dataset for NLU. The dataset is based on Flores-200 machine translation corpus. We annotated the English portion of the dataset and extended the sentence-level annotation to the remaining 203 languages covered in the corpus. Despite the simplicity of this task, our evaluation in full-supervised setting, cross-lingual transfer setting and prompting of large language model setting show that there is still a large gap between the performance of high-resource and low-resource languages when multilingual evaluation is scaled to numerous world languages. We found that languages unseen during the pre-training of multilingual language models, under-represented language families (like Nilotic and Altantic-Congo), and languages from the regions of Africa, Americas, Oceania and South East Asia, often have the lowest performance on our topic classification dataset. We hope our dataset will encourage a more inclusive evaluation of multilingual language models on a more diverse set of languages. https://github.com/dadelani/sib-200