# Kr'eyoLID From Language Identification Towards Language Mining

arXiv: https://arxiv.org/abs/2503.06547 · HTML: https://mlsanity.com/paper/2503.06547/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 15 of its 30 references that are in the map.
- First posted on arXiv 9 Mar 2025.

## Builds on (most cited first, 15 of 15)

- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [No Language Left Behind: Scaling Human-Centered Machine Translation](https://mlsanity.com/paper/2207.04672/) (2022)
- [Bag of Tricks for Efficient Text Classification](https://mlsanity.com/paper/1607.01759/) (2016)
- [Learning Word Vectors for 157 Languages](https://mlsanity.com/paper/1802.06893/) (2018)
- [Quality at a Glance: An Audit of Web-Crawled Multilingual Datasets](https://mlsanity.com/paper/2103.12028/) (2021)
- [MADLAD-400: A Multilingual And Document-Level Large Audited Dataset](https://mlsanity.com/paper/2309.04662/) (2023)
- [Towards a Cleaner Document-Oriented Multilingual Crawled Corpus](https://mlsanity.com/paper/2201.06642/) (2022)
- [Language ID in the Wild: Unexpected Challenges on the Path to a Thousand-Language Web Text Corpus](https://mlsanity.com/paper/2010.14571/) (2020)
- [An Open Dataset and Model for Language Identification](https://mlsanity.com/paper/2305.13820/) (2023)
- [AfroLID: A Neural Language Identification Tool for African Languages](https://mlsanity.com/paper/2210.11744/) (2022)
- [What a Creole Wants, What a Creole Needs](https://mlsanity.com/paper/2206.00437/) (2022)
- [Natural Language Processing with Small Feed-Forward Networks](https://mlsanity.com/paper/1708.00214/) (2017)
- [Krey`ol-MT: Building MT for Latin American, Caribbean and Colonial African Creole Languages](https://mlsanity.com/paper/2405.05376/) (2024)
- [CreoleVal: Multilingual Multitask Benchmarks for Creoles](https://mlsanity.com/paper/2310.19567/) (2023)
- [Moly'e: A Corpus-based Approach to Language Contact in Colonial France](https://mlsanity.com/paper/2408.04554/) (2024)

## Abstract (arXiv)

Automatic language identification is frequently framed as a multi-class classification problem. However, when creating digital corpora for less commonly written languages, it may be more appropriate to consider it a data mining problem. For these varieties, one knows ahead of time that the vast majority of documents are of little interest. By minimizing resources spent on classifying such documents, we can create corpora much faster and with better coverage than using established pipelines. To demonstrate the effectiveness of the language mining perspective, we introduce a new pipeline and corpora for several French-based Creoles.