# Second Language Acquisition of Neural Language Models

arXiv: https://arxiv.org/abs/2306.02920 · HTML: https://mlsanity.com/paper/2306.02920/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 43 references that are in the map.
- First posted on arXiv 5 Jun 2023.

## Builds on (most cited first, 19 of 19)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [Cross-lingual Language Model Pretraining](https://mlsanity.com/paper/1901.07291/) (2019)
- [BLiMP: The Benchmark of Linguistic Minimal Pairs for English](https://mlsanity.com/paper/1912.00582/) (2019)
- [CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Data](https://mlsanity.com/paper/1911.00359/) (2019)
- [Language Models are Multilingual Chain-of-Thought Reasoners](https://mlsanity.com/paper/2210.03057/) (2022)
- [Assessing the Ability of LSTMs to Learn Syntax-Sensitive Dependencies](https://mlsanity.com/paper/1611.01368/) (2016)
- [The Geometry of Multilingual Language Model Representations](https://mlsanity.com/paper/2205.10964/) (2022)
- [Probing Across Time: What Does RoBERTa Know and When?](https://mlsanity.com/paper/2104.07885/) (2021)
- [Cross-Linguistic Syntactic Evaluation of Word Prediction Models](https://mlsanity.com/paper/2005.00187/) (2020)
- [When is BERT Multilingual? Isolating Crucial Ingredients for Cross-lingual Transfer](https://mlsanity.com/paper/2110.14782/) (2021)
- [Can neural networks acquire a structural bias from raw linguistic data?](https://mlsanity.com/paper/2007.06761/) (2020)
- [Revisiting the poverty of the stimulus: hierarchical generalization without a hierarchical bias in recurrent neural networks](https://mlsanity.com/paper/1802.09091/) (2018)
- [Analyzing the Mono- and Cross-Lingual Pretraining Dynamics of Multilingual Language Models](https://mlsanity.com/paper/2205.11758/) (2022)
- [Pretraining with Artificial Language: Studying Transferable Knowledge in Language Models](https://mlsanity.com/paper/2203.10326/) (2022)
- [Recurrent Neural Networks in Linguistic Theory: Revisiting Pinker and Prince (1988) and the Past Tense Debate](https://mlsanity.com/paper/1807.04783/) (2018)
- [How Furiously Can Colourless Green Ideas Sleep? Sentence Acceptability in Context](https://mlsanity.com/paper/2004.00881/) (2020)
- [Assessing the Syntactic Capabilities of Transformer-based Multilingual Language Models](https://mlsanity.com/paper/2105.04688/) (2021)

## Built on by (most cited first, 8 of 8)

- [Investigating Critical Period Effects in Language Acquisition through Neural Language Models](https://mlsanity.com/paper/2407.19325/) (2024)
- [Bringing Up a Bilingual BabyLM: Investigating Multilingual Language Acquisition Using Small-Scale Models](https://mlsanity.com/paper/2603.29552/) (2026)
- [Can LLMs Simulate L2-English Dialogue? An Information-Theoretic Analysis of L1-Dependent Biases](https://mlsanity.com/paper/2502.14507/) (2025)
- [BLiSS 1.0: Evaluating Bilingual Learner Competence in Second Language Small Language Models](https://mlsanity.com/paper/2510.19419/) (2025)
- [A Study of Crosslinguistic Influence in Language Models](https://mlsanity.com/paper/2601.21587/) (2026)
- [Is Cross-Lingual Transfer in Bilingual Models Human-Like? A Study with Overlapping Word Forms in Dutch and English](https://mlsanity.com/paper/2604.07067/) (2026)
- [Dango: A Strictly L1-Only Large Language Model for Studying Second Language Acquisition](https://mlsanity.com/paper/2606.19170/) (2026)
- [Beetle: A Bilingual Model Suite for Modelling Second-Language Processing](https://mlsanity.com/paper/2609.22633/) (2026)

## Abstract (arXiv)

With the success of neural language models (LMs), their language acquisition has gained much attention. This work sheds light on the second language (L2) acquisition of LMs, while previous work has typically explored their first language (L1) acquisition. Specifically, we trained bilingual LMs with a scenario similar to human L2 acquisition and analyzed their cross-lingual transfer from linguistic perspectives. Our exploratory experiments demonstrated that the L1 pretraining accelerated their linguistic generalization in L2, and language transfer configurations (e.g., the L1 choice, and presence of parallel texts) substantially affected their generalizations. These clarify their (non-)human-like L2 acquisition in particular aspects.