mlsanityData to 14 Oct 2026

A Study of Crosslinguistic Influence in Language Models

arXiv 2601.21587 ↗First posted 29 Jan 2026cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

The sequential acquisition of languages inevitably leads to Crosslinguistic Influence (CLI), where the syntactic properties of a first language (L1) impact the processing of a second language (L2). While modern language models exhibit robust cross-lingual transfer, the exact mechanisms governing how language dominance, relative proficiency, and typological distance dictate structural interference warrant deeper investigation. In this work, we systematically investigate CLI in artificial learners by training simultaneous and sequential bilingual models across 15 typologically diverse L1s and varying the Step of Exposure (SoE), defined as the specific training step at which the L2 is introduced. Utilizing crosslinguistic structural priming, we decouple latent CLI into distinct positive and negative transfer rates. Our evaluations reveal a critical computational tradeoff: while increased L1 dominance (higher SoE) strongly amplifies the correlation between structural transfer and syntactic distance, diminished L2 proficiency bottlenecks the model's capacity for positive transfer, leaving it highly vulnerable to persistent negative interference from distant L1s. Mechanistically, we demonstrate that L1 typological proximity physically dictates the cross-lingual overlap of L2 syntactic neurons. Furthermore, we uncover that explicit priming induces a dynamic layer-wise migration of grammatical resolution to the terminal layers of the network. Through targeted causal ablations, we confirm that deep-layer attention mechanisms exclusively drive this crosslinguistic transfer by routing the L1 structural prior into the final prediction. Ultimately, our findings demonstrate that CLI in language models is not an arbitrary artifact of capacity constraints, but a structured phenomenon fundamentally governed by the interplay of language dominance and proficiency.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 39 of its 75 references that are in the map. First posted on arXiv 29 Jan 2026.

Builds on

39 of 75 references.

References · 39

  1. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
  2. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,989
  3. 2022Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 smallcited by 1,165
  4. 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,160
  5. 2019BERT Rediscovers the Classical NLP Pipelinecited by 1,059
  6. 2016Bag of Tricks for Efficient Text Classificationcited by 876
  7. 2019How multilingual is Multilingual BERT?cited by 645
  8. 2019BLiMP: The Benchmark of Linguistic Minimal Pairs for Englishcited by 564
  9. 2024Do Llamas Work in English? On the Latent Language of Multilingual Transformerscited by 281
  10. 2019Cross-Lingual Ability of Multilingual BERT: An Empirical Studycited by 182
  11. 2019Emerging Cross-lingual Structure in Pretrained Language Modelscited by 181
  12. 2023The Hydra Effect: Emergent Self-repair in Language Model Computationscited by 156
27 more
  1. 2024How do Large Language Models Handle Multilingualism?cited by 150
  2. 2025FineWeb2: One Pipeline to Scale Them All -- Adapting Pre-Training Data Processing to Every Languagecited by 143
  3. 2022The Geometry of Multilingual Language Model Representationscited by 95
  4. 2020On the Language Neutrality of Pre-trained Multilingual Representationscited by 85
  5. 2020CoSDA-ML: Multi-Lingual Code-Switching Data Augmentation for Zero-Shot Cross-Lingual NLPcited by 79
  6. 2025MultiBLiMP 1.0: A Massively Multilingual Benchmark of Linguistic Minimal Pairscited by 71
  7. 2021On the Pitfalls of Analyzing Individual Neurons in Language Modelscited by 62
  8. 2025Large Language Models Share Representations of Latent Grammatical Concepts Across Typologically Diverse Languagescited by 59
  9. 2020Cross-Linguistic Syntactic Evaluation of Word Prediction Modelscited by 55
  10. 2021A Primer on Pretrained Multilingual Language Modelscited by 49
  11. 2021Structural Persistence in Language Models: Priming as a Window into Abstract Language Representationscited by 44
  12. 2021When is BERT Multilingual? Isolating Crucial Ingredients for Cross-lingual Transfercited by 42
  13. 2024The LLM Language Network: A Neuroscientific Approach for Identifying Causally Task-Relevant Unitscited by 34
  14. 2024The Same But Different: Structural Similarities and Differences in Multilingual Language Modelingcited by 28
  15. 2021Role of Language Relatedness in Multilingual Fine-tuning of Language Models: A Case Study in Indo-Aryan Languagescited by 24
  16. 2024On the Similarity of Circuits across Languages: a Case Study on the Subject-verb Agreement Taskcited by 24
  17. 2023Romanization-based Large-scale Adaptation of Multilingual Language Modelscited by 21
  18. 2023Structural Priming Demonstrates Abstract Grammatical Representations in Multilingual Language Modelscited by 16
  19. 2025Investigating and Scaling up Code-Switching for Multilingual Language Model Pre-Trainingcited by 16
  20. 2023SLABERT Talk Pretty One Day: Modeling Second Language Acquisition with BERTcited by 15
  21. 2022Does Transliteration Help Multilingual Language Modeling?cited by 14
  22. 2022Causal Analysis of Syntactic Agreement Neurons in Multilingual Language Modelscited by 14
  23. 2023Second Language Acquisition of Neural Language Modelscited by 8
  24. 2020Understanding Cross-Lingual Syntactic Transfer in Multilingual Recurrent Neural Networkscited by 7
  25. 2025On the Acquisition of Shared Grammatical Representations in Bilingual Language Modelscited by 7
  26. 2025From Compression to Expression: A Layerwise Analysis of In-Context Learningcited by 6
  27. 2024Modeling Bilingual Sentence Processing: Evaluating RNN and Transformer Architectures for Cross-Language Structural Primingcited by 3

Built on by

Not cited yet.