# Language-specific Neurons Do Not Facilitate Cross-Lingual Transfer

arXiv: https://arxiv.org/abs/2503.17456 · HTML: https://mlsanity.com/paper/2503.17456/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 27 references that are in the map.
- First posted on arXiv 21 Mar 2025.

## Builds on (most cited first, 19 of 19)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [mT5: A massively multilingual pre-trained text-to-text transformer](https://mlsanity.com/paper/2010.11934/) (2020)
- [Cross-lingual Language Model Pretraining](https://mlsanity.com/paper/1901.07291/) (2019)
- [XNLI: Evaluating Cross-lingual Sentence Representations](https://mlsanity.com/paper/1809.05053/) (2018)
- [XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalization](https://mlsanity.com/paper/2003.11080/) (2020)
- [On the Cross-lingual Transferability of Monolingual Representations](https://mlsanity.com/paper/1910.11856/) (2019)
- [How do Large Language Models Handle Multilingualism?](https://mlsanity.com/paper/2402.18815/) (2024)
- [Analyzing Individual Neurons in Pre-trained Language Models](https://mlsanity.com/paper/2010.02695/) (2020)
- [On the Multilingual Ability of Decoder-based Pre-trained Language Models: Finding and Controlling Language-Specific Neurons](https://mlsanity.com/paper/2404.02431/) (2024)
- [BUFFET: Benchmarking Large Language Models for Few-shot Cross-lingual Transfer](https://mlsanity.com/paper/2305.14857/) (2023)
- [Style-Specific Neurons for Steering LLMs in Text Style Transfer](https://mlsanity.com/paper/2410.00593/) (2024)
- [Importance-based Neuron Allocation for Multilingual Neural Machine Translation](https://mlsanity.com/paper/2107.06569/) (2021)
- [Multilingual Knowledge Editing with Language-Agnostic Factual Neurons](https://mlsanity.com/paper/2406.16416/) (2024)
- [Tracing the Roots of Facts in Multilingual Language Models: Independent, Shared, and Transferred Knowledge](https://mlsanity.com/paper/2403.05189/) (2024)
- [Unveiling Multilinguality in Transformer Models: Exploring Language Specificity in Feed-Forward Networks](https://mlsanity.com/paper/2310.15552/) (2023)
- [Unveiling Language Competence Neurons: A Psycholinguistic Approach to Model Interpretability](https://mlsanity.com/paper/2409.15827/) (2024)

## Built on by (most cited first, 11 of 11)

- [Locate, Steer, and Improve: A Practical Survey of Actionable Mechanistic Interpretability in Large Language Models](https://mlsanity.com/paper/2601.14004/) (2026)
- [Language Arithmetics: Towards Systematic Language Neuron Identification and Manipulation](https://mlsanity.com/paper/2507.22608/) (2025)
- [What Language(s) Does Aya-23 Think In? How Multilinguality Affects Internal Language Representations](https://mlsanity.com/paper/2507.20279/) (2025)
- [Cross-Lingual Activation Steering for Multilingual Language Models](https://mlsanity.com/paper/2601.16390/) (2026)
- [From Interpretability to Performance: Optimizing Retrieval Heads for Long-Context Language Models](https://mlsanity.com/paper/2601.11020/) (2026)
- [Unveiling the Influence of Amplifying Language-Specific Neurons](https://mlsanity.com/paper/2507.22581/) (2025)
- [Linguistic Neuron Overlap Patterns to Facilitate Cross-lingual Transfer on Low-resource Languages](https://mlsanity.com/paper/2508.17078/) (2025)
- [One Model to Translate Them All? A Journey to Mount Doom for Multilingual Model Merging](https://mlsanity.com/paper/2604.02881/) (2026)
- [Sparse Subnetwork Enhancement for Underrepresented Languages in Large Language Models](https://mlsanity.com/paper/2510.13580/) (2025)
- [Tracing Multilingual Knowledge Acquisition Dynamics in Domain Adaptation: A Case Study of English-Japanese Biomedical Adaptation](https://mlsanity.com/paper/2510.12115/) (2025)
- [Distribution-aware Language Neuron Identification in Multilingual Large Language Models](https://mlsanity.com/paper/2609.10993/) (2026)

## Abstract (arXiv)

Multilingual large language models (LLMs) aim towards robust natural language understanding across diverse languages, yet their performance significantly degrades on low-resource languages. This work explores whether existing techniques to identify language-specific neurons can be leveraged to enhance cross-lingual task performance of lowresource languages. We conduct detailed experiments covering existing language-specific neuron identification techniques (such as Language Activation Probability Entropy and activation probability-based thresholding) and neuron-specific LoRA fine-tuning with models like Llama 3.1 and Mistral Nemo. We find that such neuron-specific interventions are insufficient to yield cross-lingual improvements on downstream tasks (XNLI, XQuAD) in lowresource languages. This study highlights the challenges in achieving cross-lingual generalization and provides critical insights for multilingual LLMs.