mlsanityData to 14 Oct 2026

AfroLID: A Neural Language Identification Tool for African Languages

arXiv 2210.11744 ↗First posted 21 Oct 2022cs.CL cs.LGcited by 39

Not on the map. Why?

Abstract (arXiv)

Language identification (LID) is a crucial precursor for NLP, especially for mining web data. Problematically, most of the world's 7000+ languages today are not covered by LID technologies. We address this pressing issue for Africa by introducing AfroLID, a neural LID toolkit for 517 African languages and varieties. AfroLID exploits a multi-domain web dataset manually curated from across 14 language families utilizing five orthographic systems. When evaluated on our blind Test set, AfroLID achieves 95.89 F1-score. We also compare AfroLID to five existing LID tools that each cover a small number of African languages, finding it to outperform them on most languages. We further show the utility of AfroLID in the wild by testing it on the acutely under-served Twitter domain. Finally, we offer a number of controlled case studies and perform a linguistically-motivated error analysis that allow us to both showcase AfroLID's powerful capabilities and limitations.

Key facts. Cited by 39 papers in the map (data through 14 Oct 2026). Builds on 23 of its 79 references that are in the map. First posted on arXiv 21 Oct 2022.

Builds on

23 of 79 references.

References · 23

  1. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,548
  2. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,057
  3. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
  4. 2014Sequence to Sequence Learning with Neural Networkscited by 2,986
  5. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,818
  6. 2016Bag of Tricks for Efficient Text Classificationcited by 876
  7. 2016FastText.zip: Compressing text classification modelscited by 335
  8. 2021Quality at a Glance: An Audit of Web-Crawled Multilingual Datasetscited by 243
  9. 2019WikiMatrix: Mining 135M Parallel Sentences in 1620 Language Pairs from Wikipediacited by 223
  10. 2020ARBERT & MARBERT: Deep Bidirectional Transformers for Arabiccited by 166
  11. 2019CCAligned: A Massive Collection of Cross-Lingual Web-Document Pairscited by 136
  12. 2020A Monolingual Approach to Contextualized Word Embeddings for Mid-Resource Languagescited by 131
11 more
  1. 2020Language ID in the Wild: Unexpected Challenges on the Path to a Thousand-Language Web Text Corpuscited by 77
  2. 2022NaijaSenti: A Nigerian Twitter Sentiment Corpus for Multilingual Sentiment Analysiscited by 64
  3. 2021AraT5: Text-to-Text Transformers for Arabic Language Generationcited by 63
  4. 2022Towards Afrocentric NLP for African Languages: Where We Are and Where We Can Gocited by 47
  5. 2020Mapping Languages: The Corpus of Global Language Usecited by 42
  6. 2018Automatic Language Identification in Texts: A Surveycited by 41
  7. 2019Massive vs. Curated Word Embeddings for Low-Resourced Languages. The Case of Yor`ub'a and Twicited by 33
  8. 2021Improved Language Identification Through Cross-Lingual Self-Supervised Learningcited by 20
  9. 2022Ancestor-to-Creole Transfer is Not a Walk in the Parkcited by 7
  10. 2017Improved Text Language Identification for the South African Languagescited by 4
  11. 2018A Portuguese Native Language Identification Datasetcited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 1; 2023: 8; 2024: 6; 2025: 12; 2026: 12.

39 papers.

Sort:

By area:

  1. 2023An Open Dataset and Model for Language Identificationcited by 50
  2. 2023SemEval-2023 Task 12: Sentiment Analysis for African Languages (AfriSenti-SemEval)cited by 41
  3. 2022SERENGETI: Massively Multilingual Language Models for Africacited by 40
  4. 2023GlotLID: Language Identification for Low-Resource Languagescited by 39
  5. 2023AfriQA: Cross-lingual Open-Retrieval Question Answering for African Languagescited by 23
  6. 2024GlotCC: An Open Broad-Coverage CommonCrawl Corpus and Pipeline for Minority Languagescited by 16
  7. 2025Charting the Landscape of African NLP: Mapping Progress and Shaping the Road Aheadcited by 16
  8. 2024MaskLID: Code-Switching Language Identification through Iterative Maskingcited by 11
  9. 2025Where Are We? Evaluating LLM Performance on African Languagescited by 10
  10. 2023LIMIT: Language Identification, Misidentification, and Translation using Hierarchical Models in 350+ Languagescited by 6
  11. 2023Fumbling in Babel: An Investigation into ChatGPT's Language Identification Abilitycited by 6
  12. 2024Geographically-Informed Language Identificationcited by 6
27 more
  1. 2025ConLID: Supervised Contrastive Learning for Low-Resource Language Identificationcited by 6
  2. 2026CommonLID: Re-evaluating State-of-the-Art Language Identification Performance on Web Datacited by 6
  3. 2024A Systematic Review of NLP for Ghanaian Languages: Datasets, Models, and a Research Roadmapcited by 5
  4. 2026OpenLID-v3: Improving the Precision of Closely Related Language Identification -- An Experience Reportcited by 5
  5. 2023AfroDigits: A Community-Driven Spoken Digit Dataset for African Languagescited by 4
  6. 2025Voice of a Continent: Mapping Africa's Speech Technology Frontiercited by 4
  7. 2024Code-Switched Language Identification is Harder Than You Thinkcited by 3
  8. 2025DIVERS-Bench: Evaluating Language Identification Across Domain Shifts and Code-Switchingcited by 3
  9. 2026GhanaNLP Parallel Corpora: Comprehensive Multilingual Resources for Low-Resource Ghanaian Languagescited by 3
  10. 2024From N-grams to Pre-trained Multilingual Models For Language Identificationcited by 2
  11. 2025Bridging Gaps in Natural Language Processing for Yor`ub'a: A Systematic Review of a Decade of Progress and Prospectscited by 2
  12. 2026AfriNLLB: Efficient Translation Models for African Languagescited by 2
  13. 2025Improving Informally Romanized Language Identificationcited by 1
  14. 2023UBC-DLNLP at SemEval-2023 Task 12: Impact of Transfer Learning on African Sentiment Analysiscited by 0
  15. 2025NaijaNLP: A Survey of Nigerian Low-Resource Languagescited by 0
  16. 2025Kr'eyoLID From Language Identification Towards Language Miningcited by 0
  17. 2025The Rise of AfricaNLP: A Survey of Contributions, Contributors, Community Impact, and Bibliometric Analysiscited by 0
  18. 2025Language Diversity: Evaluating Language Usage and AI Performance on African Languages in Digital Spacescited by 0
  19. 2025Opportunities and Challenges of Natural Language Processing for Low-Resource Senegalese Languages in Social Science Researchcited by 0
  20. 2026AfroScope: A Framework for Studying the Linguistic Landscape of Africacited by 0
  21. 2026How Should We Model the Probability of a Language?cited by 0
  22. 2026Sometin Beta Pass Notin: Improving Multilingual ASR for Nigerian Languages via Knowledge Distillationcited by 0
  23. 2026Temporal Simultaneity Predicts Annotation Quality in Sentiment Corporacited by 0
  24. 2026Sample-Size Scaling of the African Languages NLI Evaluationcited by 0
  25. 2026TranslatePsy-AfriSLM: High-Quality Data Scaling For Low-Resource Machine Translationcited by 0
  26. 2026Improving Language Identification for Code-Switched Utterances with Integer Linear Programmingcited by 0
  27. 2026MixLID: Token-Level Language Identification in Mixed-Language Textcited by 0