# Cross-Lingual Generalization and Compression: From Language-Specific to Shared Neurons

arXiv: https://arxiv.org/abs/2506.01629 · HTML: https://mlsanity.com/paper/2506.01629/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Jun 2025.

## Built on by (most cited first, 6 of 6)

- [When Meanings Meet: Investigating the Emergence and Quality of Shared Concept Spaces during Multilingual Language Model Training](https://mlsanity.com/paper/2601.22851/) (2026)
- [Multilingual Steering by Design: Multilingual Sparse Autoencoders and Principled Layer Selection](https://mlsanity.com/paper/2605.23036/) (2026)
- [CrossHallu: Do Hallucination Signals Generalize Across Languages and Domains in Large Language Model's Internals?](https://mlsanity.com/paper/2607.04029/) (2026)
- [Copy First, Translate Later: Interpreting Translation Dynamics in Multilingual Pretraining](https://mlsanity.com/paper/2604.17633/) (2026)
- [Cross-Lingual Alignment Without Joint Training: Do Monolingual Language Models Converge on Universal Representations?](https://mlsanity.com/paper/2608.27115/) (2026)
- [Target-Language Generation in Multilingual Models: Activation Steering and Optimal Control](https://mlsanity.com/paper/2609.16967/) (2026)

## Abstract (arXiv)

Multilingual language models (MLLMs) have demonstrated remarkable abilities to transfer knowledge across languages, despite being trained without explicit cross-lingual supervision. We analyze the parameter spaces of three MLLMs to study how their representations evolve during pre-training, observing patterns consistent with compression: models initially form language-specific representations, which gradually converge into cross-lingual abstractions as training progresses. Through probing experiments, we observe a clear transition from uniform language identification capabilities across layers to more specialized layer functions. For deeper analysis, we focus on neurons that encode distinct semantic concepts. By tracing their development during pre-training, we show how they gradually align across languages. Notably, we identify specific neurons that emerge as increasingly reliable predictors for the same concepts across languages.