# Crosslingual Structural Priming and the Pre-Training Dynamics of Bilingual Language Models

arXiv: https://arxiv.org/abs/2310.07929 · HTML: https://mlsanity.com/paper/2310.07929/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 9 of its 20 references that are in the map.
- First posted on arXiv 11 Oct 2023.

## Builds on (most cited first, 9 of 9)

- [XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalization](https://mlsanity.com/paper/2003.11080/) (2020)
- [On the Cross-lingual Transferability of Monolingual Representations](https://mlsanity.com/paper/1910.11856/) (2019)
- [Beto, Bentz, Becas: The Surprising Cross-Lingual Effectiveness of BERT](https://mlsanity.com/paper/1904.09077/) (2019)
- [Language Models are Few-shot Multilingual Learners](https://mlsanity.com/paper/2109.07684/) (2021)
- [Language Contamination Helps Explain the Cross-lingual Capabilities of English Pretrained Models](https://mlsanity.com/paper/2204.08110/) (2022)
- [Cross-Lingual Natural Language Generation via Pre-Training](https://mlsanity.com/paper/1909.10481/) (2019)
- [Structural Persistence in Language Models: Priming as a Window into Abstract Language Representations](https://mlsanity.com/paper/2109.14989/) (2021)
- [Using Priming to Uncover the Organization of Syntactic Representations in Neural Language Models](https://mlsanity.com/paper/1909.10579/) (2019)
- [Neural reality of argument structure constructions](https://mlsanity.com/paper/2202.12246/) (2022)

## Abstract (arXiv)

Do multilingual language models share abstract grammatical representations across languages, and if so, when do these develop? Following Sinclair et al. (2022), we use structural priming to test for abstract grammatical representations with causal effects on model outputs. We extend the approach to a Dutch-English bilingual setting, and we evaluate a Dutch-English language model during pre-training. We find that crosslingual structural priming effects emerge early after exposure to the second language, with less than 1M tokens of data in that language. We discuss implications for data contamination, low-resource transfer, and how abstract grammatical representations emerge in multilingual models.