mlsanityData to 14 Oct 2026

AfroScope: A Framework for Studying the Linguistic Landscape of Africa

arXiv 2601.13346 ↗First posted 19 Jan 2026cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

Language Identification (LID), the task of determining the language of a given text, is a fundamental preprocessing step that shapes the reliability of downstream NLP applications. While recent work has expanded African LID, existing systems remain limited in both language coverage and fine-grained discrimination among closely related languages and varieties. We introduce AfroScope, a unified framework for African LID that includes AfroScope-Data, a dataset covering 640 languages, and AfroScope-Models, a suite of strong LID models with broad African language coverage. To address persistent confusions among closely related languages, we propose a hierarchical classification approach that leverages AfroScope-Mirror, a specialized embedding model for targeted disambiguation, improving macro-F1 by 1.57 points on the confusable subset compared to our best base model. We further analyze cross-lingual transfer and domain effects, showing how language-family structure, script compatibility, and domain coverage shape LID performance. We position African LID as an enabling technology for large-scale measurement of Africa's linguistic landscape in digital text, and release AfroScope-Data and AfroScope-Models online.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 41 of its 55 references that are in the map. First posted on arXiv 19 Jan 2026.

Builds on

41 of 55 references.

References · 41

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  3. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
  4. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,818
  5. 2022No Language Left Behind: Scaling Human-Centered Machine Translationcited by 1,312
  6. 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,160
  7. 2016Bag of Tricks for Efficient Text Classificationcited by 876
  8. 2024DataComp-LM: In search of the next generation of training sets for language modelscited by 398
  9. 2016FastText.zip: Compressing text classification modelscited by 335
  10. 2023Studying Large Language Model Generalization with Influence Functionscited by 326
  11. 2019Massively Multilingual Neural Machine Translation in the Wild: Findings and Challengescited by 257
  12. 2021Quality at a Glance: An Audit of Web-Crawled Multilingual Datasetscited by 243
29 more
  1. 2019WikiMatrix: Mining 135M Parallel Sentences in 1620 Language Pairs from Wikipediacited by 223
  2. 2025FineWeb2: One Pipeline to Scale Them All -- Adapting Pre-Training Data Processing to Every Languagecited by 143
  3. 2022Impact of Pretraining Term Frequencies on Few-Shot Reasoningcited by 142
  4. 2022A Few Thousand Translations Go a Long Way! Leveraging Pre-trained Models for African News Translationcited by 98
  5. 2021Fast, Effective, and Self-Supervised: Transforming Masked Language Models into Universal Lexical and Sentence Encoderscited by 88
  6. 2020Language ID in the Wild: Unexpected Challenges on the Path to a Thousand-Language Web Text Corpuscited by 77
  7. 2017Racial Disparity in Natural Language Processing: A Case Study of Social Media African-American Englishcited by 71
  8. 2024A New Massive Multilingual Dataset for High-Performance Language Technologiescited by 56
  9. 2023An Open Dataset and Model for Language Identificationcited by 50
  10. 2024IrokoBench: A New Benchmark for African Languages in the Age of Large Language Modelscited by 43
  11. 2022SERENGETI: Massively Multilingual Language Models for Africacited by 40
  12. 2022AfroLID: A Neural Language Identification Tool for African Languagescited by 39
  13. 2019Massive vs. Curated Word Embeddings for Low-Resourced Languages. The Case of Yor`ub'a and Twicited by 33
  14. 2022Bloom Library: Multimodal Datasets in 300+ Languages for a Variety of Downstream Taskscited by 28
  15. 2024Opportunities and Challenges of Large Language Models for Low-Resource Languages in Humanities Researchcited by 27
  16. 2025ATLAS: Adaptive Transfer Scaling Laws for Multilingual Pretraining, Finetuning, and Decoding the Curse of Multilingualitycited by 22
  17. 2025Charting the Landscape of African NLP: Mapping Progress and Shaping the Road Aheadcited by 16
  18. 2024Cheetah: Natural Language Generation for 517 African Languagescited by 15
  19. 2024Correcting FLORES Evaluation Dataset for Four African Languagescited by 13
  20. 2025SMOL: Professionally translated parallel data for 115 under-represented languagescited by 12
  21. 2023Multilingual self-supervised speech representations improve the speech recognition of low-resource African languages with codeswitchingcited by 11
  22. 2025Data Quality Issues in Multilingual Speech Datasets: The Need for Sociolinguistic Awareness and Proactive Language Planningcited by 11
  23. 2025Where Are We? Evaluating LLM Performance on African Languagescited by 10
  24. 2025The State of Large Language Models for African Languages: Progress and Challengescited by 8
  25. 2023LIMIT: Language Identification, Misidentification, and Translation using Hierarchical Models in 350+ Languagescited by 6
  26. 2025ConLID: Supervised Contrastive Learning for Low-Resource Language Identificationcited by 6
  27. 2017Improved Text Language Identification for the South African Languagescited by 4
  28. 2025Voice of a Continent: Mapping Africa's Speech Technology Frontiercited by 4
  29. 2025DIVERS-Bench: Evaluating Language Identification Across Domain Shifts and Code-Switchingcited by 3

Built on by

Not cited yet.