# Same Neurons, Different Languages: Probing Morphosyntax in Multilingual Pre-trained Models

arXiv: https://arxiv.org/abs/2205.02023 · HTML: https://mlsanity.com/paper/2205.02023/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 26 references that are in the map.
- First posted on arXiv 4 May 2022.

## Builds on (most cited first, 17 of 17)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [mT5: A massively multilingual pre-trained text-to-text transformer](https://mlsanity.com/paper/2010.11934/) (2020)
- [Multilingual Denoising Pre-training for Neural Machine Translation](https://mlsanity.com/paper/2001.08210/) (2020)
- [How multilingual is Multilingual BERT?](https://mlsanity.com/paper/1906.01502/) (2019)
- [Beto, Bentz, Becas: The Surprising Cross-Lingual Effectiveness of BERT](https://mlsanity.com/paper/1904.09077/) (2019)
- [Probing Pretrained Language Models for Lexical Semantics](https://mlsanity.com/paper/2010.05731/) (2020)
- [What Is One Grain of Sand in the Desert? Analyzing Individual Neurons in Deep NLP Models](https://mlsanity.com/paper/1812.09355/) (2018)
- [Multilingual Alignment of Contextual Word Representations](https://mlsanity.com/paper/2002.03518/) (2020)
- [Zero-Shot Cross-Lingual Transfer with Meta Learning](https://mlsanity.com/paper/2003.02739/) (2020)
- [On the Pitfalls of Analyzing Individual Neurons in Language Models](https://mlsanity.com/paper/2110.07483/) (2021)
- [Inducing Language-Agnostic Multilingual Representations](https://mlsanity.com/paper/2008.09112/) (2020)
- [Intrinsic Probing through Dimension Selection](https://mlsanity.com/paper/2010.02812/) (2020)
- [UniMorph 2.0: Universal Morphology](https://mlsanity.com/paper/1810.11101/) (2018)
- [Few-Shot Cross-Lingual Stance Detection with Sentiment-Based Pre-Training](https://mlsanity.com/paper/2109.06050/) (2021)
- [Marrying Universal Dependencies and Universal Morphology](https://mlsanity.com/paper/1810.06743/) (2018)
- [A Latent-Variable Model for Intrinsic Probing](https://mlsanity.com/paper/2201.08214/) (2022)

## Built on by (most cited first, 14 of 14)

- [Modular Deep Learning](https://mlsanity.com/paper/2302.11529/) (2023)
- [On the Multilingual Ability of Decoder-based Pre-trained Language Models: Finding and Controlling Language-Specific Neurons](https://mlsanity.com/paper/2404.02431/) (2024)
- [MultiBLiMP 1.0: A Massively Multilingual Benchmark of Linguistic Minimal Pairs](https://mlsanity.com/paper/2504.02768/) (2025)
- [Separating Tongue from Thought: Activation Patching Reveals Language-Agnostic Concept Representations in Transformers](https://mlsanity.com/paper/2411.08745/) (2024)
- [Phylogeny-Inspired Adaptation of Multilingual Models to New Languages](https://mlsanity.com/paper/2205.09634/) (2022)
- [Causal Analysis of Syntactic Agreement Neurons in Multilingual Language Models](https://mlsanity.com/paper/2210.14328/) (2022)
- [Colexifications for Bootstrapping Cross-lingual Datasets: The Case of Phonology, Concreteness, and Affectiveness](https://mlsanity.com/paper/2306.02646/) (2023)
- [Assessment of Pre-Trained Models Across Languages and Grammars](https://mlsanity.com/paper/2309.11165/) (2023)
- [Neuron-Level Interventions for Gendered and Gender-Neutral Generation in Language Models](https://mlsanity.com/paper/2605.30717/) (2026)
- [Multilingual Text Representation](https://mlsanity.com/paper/2309.00949/) (2023)
- [Polysemy of Synthetic Neurons Towards a New Type of Explanatory Categorical Vector Spaces](https://mlsanity.com/paper/2505.07831/) (2025)
- [Brain-LLM Alignment Tracks Training Data, Not Typology](https://mlsanity.com/paper/2605.23032/) (2026)
- [Shared Circuits for Shared Grammar: Tracing Subject-Verb Agreement Across Languages](https://mlsanity.com/paper/2608.18545/) (2026)
- [Hallucination Neurons and Where to Find Them: An Investigation into the existence of Hallucination Neurons](https://mlsanity.com/paper/2609.29781/) (2026)

## Abstract (arXiv)

The success of multilingual pre-trained models is underpinned by their ability to learn representations shared by multiple languages even in absence of any explicit supervision. However, it remains unclear how these models learn to generalise across languages. In this work, we conjecture that multilingual pre-trained models can derive language-universal abstractions about grammar. In particular, we investigate whether morphosyntactic information is encoded in the same subset of neurons in different languages. We conduct the first large-scale empirical study over 43 languages and 14 morphosyntactic categories with a state-of-the-art neuron-level probe. Our findings show that the cross-lingual overlap between neurons is significant, but its extent may vary across categories and depends on language proximity and pre-training data size.