# AfroLID: A Neural Language Identification Tool for African Languages

arXiv: https://arxiv.org/abs/2210.11744 · HTML: https://mlsanity.com/paper/2210.11744/

## Key facts

- Cited by 39 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 79 references that are in the map.
- First posted on arXiv 21 Oct 2022.

## Builds on (most cited first, 23 of 23)

- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [Sequence to Sequence Learning with Neural Networks](https://mlsanity.com/paper/1409.3215/) (2014)
- [mT5: A massively multilingual pre-trained text-to-text transformer](https://mlsanity.com/paper/2010.11934/) (2020)
- [Bag of Tricks for Efficient Text Classification](https://mlsanity.com/paper/1607.01759/) (2016)
- [FastText.zip: Compressing text classification models](https://mlsanity.com/paper/1612.03651/) (2016)
- [Quality at a Glance: An Audit of Web-Crawled Multilingual Datasets](https://mlsanity.com/paper/2103.12028/) (2021)
- [WikiMatrix: Mining 135M Parallel Sentences in 1620 Language Pairs from Wikipedia](https://mlsanity.com/paper/1907.05791/) (2019)
- [ARBERT & MARBERT: Deep Bidirectional Transformers for Arabic](https://mlsanity.com/paper/2101.01785/) (2020)
- [CCAligned: A Massive Collection of Cross-Lingual Web-Document Pairs](https://mlsanity.com/paper/1911.06154/) (2019)
- [A Monolingual Approach to Contextualized Word Embeddings for Mid-Resource Languages](https://mlsanity.com/paper/2006.06202/) (2020)
- [Language ID in the Wild: Unexpected Challenges on the Path to a Thousand-Language Web Text Corpus](https://mlsanity.com/paper/2010.14571/) (2020)
- [NaijaSenti: A Nigerian Twitter Sentiment Corpus for Multilingual Sentiment Analysis](https://mlsanity.com/paper/2201.08277/) (2022)
- [AraT5: Text-to-Text Transformers for Arabic Language Generation](https://mlsanity.com/paper/2109.12068/) (2021)
- [Towards Afrocentric NLP for African Languages: Where We Are and Where We Can Go](https://mlsanity.com/paper/2203.08351/) (2022)
- [Mapping Languages: The Corpus of Global Language Use](https://mlsanity.com/paper/2004.00798/) (2020)
- [Automatic Language Identification in Texts: A Survey](https://mlsanity.com/paper/1804.08186/) (2018)
- [Massive vs. Curated Word Embeddings for Low-Resourced Languages. The Case of Yor`ub'a and Twi](https://mlsanity.com/paper/1912.02481/) (2019)
- [Improved Language Identification Through Cross-Lingual Self-Supervised Learning](https://mlsanity.com/paper/2107.04082/) (2021)
- [Ancestor-to-Creole Transfer is Not a Walk in the Park](https://mlsanity.com/paper/2206.04371/) (2022)
- [Improved Text Language Identification for the South African Languages](https://mlsanity.com/paper/1711.00247/) (2017)
- [A Portuguese Native Language Identification Dataset](https://mlsanity.com/paper/1804.11346/) (2018)

## Built on by (most cited first, 30 of 39)

- [An Open Dataset and Model for Language Identification](https://mlsanity.com/paper/2305.13820/) (2023)
- [SemEval-2023 Task 12: Sentiment Analysis for African Languages (AfriSenti-SemEval)](https://mlsanity.com/paper/2304.06845/) (2023)
- [SERENGETI: Massively Multilingual Language Models for Africa](https://mlsanity.com/paper/2212.10785/) (2022)
- [GlotLID: Language Identification for Low-Resource Languages](https://mlsanity.com/paper/2310.16248/) (2023)
- [AfriQA: Cross-lingual Open-Retrieval Question Answering for African Languages](https://mlsanity.com/paper/2305.06897/) (2023)
- [GlotCC: An Open Broad-Coverage CommonCrawl Corpus and Pipeline for Minority Languages](https://mlsanity.com/paper/2410.23825/) (2024)
- [Charting the Landscape of African NLP: Mapping Progress and Shaping the Road Ahead](https://mlsanity.com/paper/2505.21315/) (2025)
- [MaskLID: Code-Switching Language Identification through Iterative Masking](https://mlsanity.com/paper/2406.06263/) (2024)
- [Where Are We? Evaluating LLM Performance on African Languages](https://mlsanity.com/paper/2502.19582/) (2025)
- [LIMIT: Language Identification, Misidentification, and Translation using Hierarchical Models in 350+ Languages](https://mlsanity.com/paper/2305.14263/) (2023)
- [Fumbling in Babel: An Investigation into ChatGPT's Language Identification Ability](https://mlsanity.com/paper/2311.09696/) (2023)
- [Geographically-Informed Language Identification](https://mlsanity.com/paper/2403.09892/) (2024)
- [ConLID: Supervised Contrastive Learning for Low-Resource Language Identification](https://mlsanity.com/paper/2506.15304/) (2025)
- [CommonLID: Re-evaluating State-of-the-Art Language Identification Performance on Web Data](https://mlsanity.com/paper/2601.18026/) (2026)
- [A Systematic Review of NLP for Ghanaian Languages: Datasets, Models, and a Research Roadmap](https://mlsanity.com/paper/2405.06818/) (2024)
- [OpenLID-v3: Improving the Precision of Closely Related Language Identification -- An Experience Report](https://mlsanity.com/paper/2602.13139/) (2026)
- [AfroDigits: A Community-Driven Spoken Digit Dataset for African Languages](https://mlsanity.com/paper/2303.12582/) (2023)
- [Voice of a Continent: Mapping Africa's Speech Technology Frontier](https://mlsanity.com/paper/2505.18436/) (2025)
- [Code-Switched Language Identification is Harder Than You Think](https://mlsanity.com/paper/2402.01505/) (2024)
- [DIVERS-Bench: Evaluating Language Identification Across Domain Shifts and Code-Switching](https://mlsanity.com/paper/2509.17768/) (2025)
- [GhanaNLP Parallel Corpora: Comprehensive Multilingual Resources for Low-Resource Ghanaian Languages](https://mlsanity.com/paper/2603.13793/) (2026)
- [From N-grams to Pre-trained Multilingual Models For Language Identification](https://mlsanity.com/paper/2410.08728/) (2024)
- [Bridging Gaps in Natural Language Processing for Yor`ub'a: A Systematic Review of a Decade of Progress and Prospects](https://mlsanity.com/paper/2502.17364/) (2025)
- [AfriNLLB: Efficient Translation Models for African Languages](https://mlsanity.com/paper/2602.09373/) (2026)
- [Improving Informally Romanized Language Identification](https://mlsanity.com/paper/2504.21540/) (2025)
- [UBC-DLNLP at SemEval-2023 Task 12: Impact of Transfer Learning on African Sentiment Analysis](https://mlsanity.com/paper/2304.11256/) (2023)
- [NaijaNLP: A Survey of Nigerian Low-Resource Languages](https://mlsanity.com/paper/2502.19784/) (2025)
- [Kr'eyoLID From Language Identification Towards Language Mining](https://mlsanity.com/paper/2503.06547/) (2025)
- [The Rise of AfricaNLP: A Survey of Contributions, Contributors, Community Impact, and Bibliometric Analysis](https://mlsanity.com/paper/2509.25477/) (2025)
- [Language Diversity: Evaluating Language Usage and AI Performance on African Languages in Digital Spaces](https://mlsanity.com/paper/2512.01557/) (2025)

## Abstract (arXiv)

Language identification (LID) is a crucial precursor for NLP, especially for mining web data. Problematically, most of the world's 7000+ languages today are not covered by LID technologies. We address this pressing issue for Africa by introducing AfroLID, a neural LID toolkit for 517 African languages and varieties. AfroLID exploits a multi-domain web dataset manually curated from across 14 language families utilizing five orthographic systems. When evaluated on our blind Test set, AfroLID achieves 95.89 F1-score. We also compare AfroLID to five existing LID tools that each cover a small number of African languages, finding it to outperform them on most languages. We further show the utility of AfroLID in the wild by testing it on the acutely under-served Twitter domain. Finally, we offer a number of controlled case studies and perform a linguistically-motivated error analysis that allow us to both showcase AfroLID's powerful capabilities and limitations.