# SLABERT Talk Pretty One Day: Modeling Second Language Acquisition with BERT

arXiv: https://arxiv.org/abs/2305.19589 · HTML: https://mlsanity.com/paper/2305.19589/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 69 references that are in the map.
- First posted on arXiv 31 May 2023.

## Builds on (most cited first, 14 of 14)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [HuggingFace's Transformers: State-of-the-art Natural Language Processing](https://mlsanity.com/paper/1910.03771/) (2019)
- [XNLI: Evaluating Cross-lingual Sentence Representations](https://mlsanity.com/paper/1809.05053/) (2018)
- [BLiMP: The Benchmark of Linguistic Minimal Pairs for English](https://mlsanity.com/paper/1912.00582/) (2019)
- [Word Translation Without Parallel Data](https://mlsanity.com/paper/1710.04087/) (2017)
- [What BERT is not: Lessons from a new suite of psycholinguistic diagnostics for language models](https://mlsanity.com/paper/1907.13528/) (2019)
- [Beto, Bentz, Becas: The Surprising Cross-Lingual Effectiveness of BERT](https://mlsanity.com/paper/1904.09077/) (2019)
- [Negated and Misprimed Probes for Pretrained Language Models: Birds Can Talk, But Cannot Fly](https://mlsanity.com/paper/1911.03343/) (2019)
- [Emerging Cross-lingual Structure in Pretrained Language Models](https://mlsanity.com/paper/1911.01464/) (2019)
- [A robust self-learning method for fully unsupervised cross-lingual mappings of word embeddings](https://mlsanity.com/paper/1805.06297/) (2018)
- [A Survey Of Cross-lingual Word Embedding Models](https://mlsanity.com/paper/1706.04902/) (2017)
- [Learning to Understand Child-directed and Adult-directed Speech](https://mlsanity.com/paper/2005.02721/) (2020)
- [Reconstructing Native Language Typology from Foreign Language Usage](https://mlsanity.com/paper/1404.6312/) (2014)

## Built on by (most cited first, 15 of 15)

- [A Principled Framework for Evaluating on Typologically Diverse Languages](https://mlsanity.com/paper/2407.05022/) (2024)
- [BabyBabelLM: A Multilingual Benchmark of Developmentally Plausible Training Data](https://mlsanity.com/paper/2510.10159/) (2025)
- [IPA-CHILDES & G2P+: Feature-Rich Resources for Cross-Lingual Phonology and Phonemic Language Modeling](https://mlsanity.com/paper/2504.03036/) (2025)
- [BAMBINO-LM: (Bilingual-)Human-Inspired Continual Pretraining of BabyLM](https://mlsanity.com/paper/2406.11418/) (2024)
- [Bringing Up a Bilingual BabyLM: Investigating Multilingual Language Acquisition Using Small-Scale Models](https://mlsanity.com/paper/2603.29552/) (2026)
- [Enhancing Linguistic Competence of Language Models through Pre-training with Language Learning Tasks](https://mlsanity.com/paper/2601.03448/) (2026)
- [What is "Typological Diversity" in NLP?](https://mlsanity.com/paper/2402.04222/) (2024)
- [Can LLMs Simulate L2-English Dialogue? An Information-Theoretic Analysis of L1-Dependent Biases](https://mlsanity.com/paper/2502.14507/) (2025)
- [Learning from Child-Directed Speech in Two-Language Scenarios: A French-English Case Study](https://mlsanity.com/paper/2603.12906/) (2026)
- [BLiSS 1.0: Evaluating Bilingual Learner Competence in Second Language Small Language Models](https://mlsanity.com/paper/2510.19419/) (2025)
- [A Study of Crosslinguistic Influence in Language Models](https://mlsanity.com/paper/2601.21587/) (2026)
- [Dango: A Strictly L1-Only Large Language Model for Studying Second Language Acquisition](https://mlsanity.com/paper/2606.19170/) (2026)
- [Beetle: A Bilingual Model Suite for Modelling Second-Language Processing](https://mlsanity.com/paper/2609.22633/) (2026)
- [A retrospective analysis on the use of LLMs to study infant syntax learning](https://mlsanity.com/paper/2609.26539/) (2026)
- [Cross-Linguistic Effects in Bilingual Phoneme BabyLMs](https://mlsanity.com/paper/2609.37121/) (2026)

## Abstract (arXiv)

Second language acquisition (SLA) research has extensively studied cross-linguistic transfer, the influence of linguistic structure of a speaker's native language [L1] on the successful acquisition of a foreign language [L2]. Effects of such transfer can be positive (facilitating acquisition) or negative (impeding acquisition). We find that NLP literature has not given enough attention to the phenomenon of negative transfer. To understand patterns of both positive and negative transfer between L1 and L2, we model sequential second language acquisition in LMs. Further, we build a Mutlilingual Age Ordered CHILDES (MAO-CHILDES) -- a dataset consisting of 5 typologically diverse languages, i.e., German, French, Polish, Indonesian, and Japanese -- to understand the degree to which native Child-Directed Speech (CDS) [L1] can help or conflict with English language acquisition [L2]. To examine the impact of native CDS, we use the TILT-based cross lingual transfer learning approach established by Papadimitriou and Jurafsky (2020) and find that, as in human SLA, language family distance predicts more negative transfer. Additionally, we find that conversational speech data shows greater facilitation for language acquisition than scripted speech data. Our findings call for further research using our novel Transformer-based SLA models and we would like to encourage it by releasing our code, data, and models.