# On Language Models for Creoles

arXiv: https://arxiv.org/abs/2109.06074 · HTML: https://mlsanity.com/paper/2109.06074/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 67 references that are in the map.
- First posted on arXiv 13 Sep 2021.

## Builds on (most cited first, 24 of 24)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [HuggingFace's Transformers: State-of-the-art Natural Language Processing](https://mlsanity.com/paper/1910.03771/) (2019)
- [WILDS: A Benchmark of in-the-Wild Distribution Shifts](https://mlsanity.com/paper/2012.07421/) (2020)
- [Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalization](https://mlsanity.com/paper/1911.08731/) (2019)
- [TinyBERT: Distilling BERT for Natural Language Understanding](https://mlsanity.com/paper/1909.10351/) (2019)
- [The State and Fate of Linguistic Diversity and Inclusion in the NLP World](https://mlsanity.com/paper/2004.09095/) (2020)
- [What Neural Networks Memorize and Why: Discovering the Long Tail via Influence Estimation](https://mlsanity.com/paper/2008.03703/) (2020)
- [Fairness Without Demographics in Repeated Loss Minimization](https://mlsanity.com/paper/1806.08010/) (2018)
- [Universal Dependencies v2: An Evergrowing Multilingual Treebank Collection](https://mlsanity.com/paper/2004.10643/) (2020)
- [BERT has a Mouth, and It Must Speak: BERT as a Markov Random Field Language Model](https://mlsanity.com/paper/1902.04094/) (2019)
- [MasakhaNER: Named Entity Recognition for African Languages](https://mlsanity.com/paper/2103.11811/) (2021)
- [Distributionally Robust Language Modeling](https://mlsanity.com/paper/1909.02060/) (2019)
- [Out-of-domain Detection for Natural Language Understanding in Dialog Systems](https://mlsanity.com/paper/1909.03862/) (2019)
- [Massive Choice, Ample Tasks (MaChAmp): A Toolkit for Multi-task Learning in NLP](https://mlsanity.com/paper/2005.14672/) (2020)
- [Mind Your Inflections! Improving NLP for Non-Standard Englishes with Base-Inflection Encoding](https://mlsanity.com/paper/2004.14870/) (2020)
- [Challenges of Computational Processing of Code-Switching](https://mlsanity.com/paper/1610.02213/) (2016)
- [Semantic Enrichment of Nigerian Pidgin English for Contextual Sentiment Classification](https://mlsanity.com/paper/2003.12450/) (2020)
- [PidginUNMT: Unsupervised Neural Machine Translation from West African Pidgin to English](https://mlsanity.com/paper/1912.03444/) (2019)
- [Universal Dependencies Parsing for Colloquial Singaporean English](https://mlsanity.com/paper/1705.06463/) (2017)
- [NaijaNER : Comprehensive Named Entity Recognition for 5 Nigerian Languages](https://mlsanity.com/paper/2105.00810/) (2021)
- [Towards Supervised and Unsupervised Neural Machine Translation Baselines for Nigerian Pidgin](https://mlsanity.com/paper/2003.12660/) (2020)
- [Towards End-to-End Training of Automatic Speech Recognition for Nigerian Pidgin](https://mlsanity.com/paper/2010.11123/) (2020)
- [Developing a concept-level knowledge base for sentiment analysis in Singlish](https://mlsanity.com/paper/1707.04408/) (2017)

## Built on by (most cited first, 17 of 17)

- [Challenges and Strategies in Cross-Cultural NLP](https://mlsanity.com/paper/2203.10020/) (2022)
- [Language Model Tokenizers Introduce Unfairness Between Languages](https://mlsanity.com/paper/2305.15425/) (2023)
- [Prompting Multilingual Large Language Models to Generate Code-Mixed Texts: The Case of South East Asian Languages](https://mlsanity.com/paper/2303.13592/) (2023)
- [Beyond Perplexity: Multi-dimensional Safety Evaluation of LLM Compression](https://mlsanity.com/paper/2407.04965/) (2024)
- [What a Creole Wants, What a Creole Needs](https://mlsanity.com/paper/2206.00437/) (2022)
- [A Material Lens on Coloniality in NLP](https://mlsanity.com/paper/2311.08391/) (2023)
- [Evaluating Distributional Distortion in Neural Language Modeling](https://mlsanity.com/paper/2203.12788/) (2022)
- [JamPatoisNLI: A Jamaican Patois Natural Language Inference Dataset](https://mlsanity.com/paper/2212.03419/) (2022)
- [CreoleVal: Multilingual Multitask Benchmarks for Creoles](https://mlsanity.com/paper/2310.19567/) (2023)
- [How Good are Commercial Large Language Models on African Languages?](https://mlsanity.com/paper/2305.06530/) (2023)
- [Ancestor-to-Creole Transfer is Not a Walk in the Park](https://mlsanity.com/paper/2206.04371/) (2022)
- [Low-Resource Cross-Lingual Adaptive Training for Nigerian Pidgin](https://mlsanity.com/paper/2307.00382/) (2023)
- [Modeling Orthographic Variation Improves NLP Performance for Nigerian Pidgin](https://mlsanity.com/paper/2404.18264/) (2024)
- [Designing Speech Technologies for Australian Aboriginal English: Opportunities, Risks and Participation](https://mlsanity.com/paper/2503.03186/) (2025)
- [Limitations of Religious Data and the Importance of the Target Domain: Towards Machine Translation for Guinea-Bissau Creole](https://mlsanity.com/paper/2504.02674/) (2025)
- [Guylingo: The Republic of Guyana Creole Corpora](https://mlsanity.com/paper/2405.03832/) (2024)
- [Evaluating Cultural Awareness of LLMs for Haitian Creole](https://mlsanity.com/paper/2609.31506/) (2026)

## Abstract (arXiv)

Creole languages such as Nigerian Pidgin English and Haitian Creole are under-resourced and largely ignored in the NLP literature. Creoles typically result from the fusion of a foreign language with multiple local languages, and what grammatical and lexical features are transferred to the creole is a complex process. While creoles are generally stable, the prominence of some features may be much stronger with certain demographics or in some linguistic situations. This paper makes several contributions: We collect existing corpora and release models for Haitian Creole, Nigerian Pidgin English, and Singaporean Colloquial English. We evaluate these models on intrinsic and extrinsic tasks. Motivated by the above literature, we compare standard language models with distributionally robust ones and find that, somewhat surprisingly, the standard language models are superior to the distributionally robust ones. We investigate whether this is an effect of over-parameterization or relative distributional stability, and find that the difference persists in the absence of over-parameterization, and that drift is limited, confirming the relative stability of creole languages.