# Ancestor-to-Creole Transfer is Not a Walk in the Park

arXiv: https://arxiv.org/abs/2206.04371 · HTML: https://mlsanity.com/paper/2206.04371/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 29 references that are in the map.
- First posted on arXiv 9 Jun 2022.

## Builds on (most cited first, 12 of 12)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation](https://mlsanity.com/paper/1609.08144/) (2016)
- [Cross-Lingual Ability of Multilingual BERT: An Empirical Study](https://mlsanity.com/paper/1912.07840/) (2019)
- [MasakhaNER: Named Entity Recognition for African Languages](https://mlsanity.com/paper/2103.11811/) (2021)
- [When Being Unseen from mBERT is just the Beginning: Handling New Languages With Multilingual Language Models](https://mlsanity.com/paper/2010.12858/) (2020)
- [UNKs Everywhere: Adapting Multilingual Language Models to New Scripts](https://mlsanity.com/paper/2012.15562/) (2020)
- [Massive Choice, Ample Tasks (MaChAmp): A Toolkit for Multi-task Learning in NLP](https://mlsanity.com/paper/2005.14672/) (2020)
- [When is BERT Multilingual? Isolating Crucial Ingredients for Cross-lingual Transfer](https://mlsanity.com/paper/2110.14782/) (2021)
- [On Language Models for Creoles](https://mlsanity.com/paper/2109.06074/) (2021)

## Built on by (most cited first, 7 of 7)

- [AfroLID: A Neural Language Identification Tool for African Languages](https://mlsanity.com/paper/2210.11744/) (2022)
- [JamPatoisNLI: A Jamaican Patois Natural Language Inference Dataset](https://mlsanity.com/paper/2212.03419/) (2022)
- [CreoleVal: Multilingual Multitask Benchmarks for Creoles](https://mlsanity.com/paper/2310.19567/) (2023)
- [Guylingo: The Republic of Guyana Creole Corpora](https://mlsanity.com/paper/2405.03832/) (2024)
- [Moly'e: A Corpus-based Approach to Language Contact in Colonial France](https://mlsanity.com/paper/2408.04554/) (2024)
- [Limited-Resource Adapters Are Regularizers, Not Linguists](https://mlsanity.com/paper/2505.24525/) (2025)
- [The University of Melbourne WMT 2026 CreoleMT Submission: A Domain-Balanced Approach to Low-Resource Pacific Creole Machine Translation](https://mlsanity.com/paper/2609.13615/) (2026)

## Abstract (arXiv)

We aim to learn language models for Creole languages for which large volumes of data are not readily available, and therefore explore the potential transfer from ancestor languages (the 'Ancestry Transfer Hypothesis'). We find that standard transfer methods do not facilitate ancestry transfer. Surprisingly, different from other non-Creole languages, a very distinct two-phase pattern emerges for Creoles: As our training losses plateau, and language models begin to overfit on their source languages, perplexity on the Creoles drop. We explore if this compression phase can lead to practically useful language models (the 'Ancestry Bottleneck Hypothesis'), but also falsify this. Moreover, we show that Creoles even exhibit this two-phase pattern even when training on random, unrelated languages. Thus Creoles seem to be typological outliers and we speculate whether there is a link between the two observations.