# Beyond the Rosetta Stone: Unification Forces in Generalization Dynamics

arXiv: https://arxiv.org/abs/2508.11017 · HTML: https://mlsanity.com/paper/2508.11017/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 36 of its 48 references that are in the map.
- First posted on arXiv 14 Aug 2025.

## Builds on (most cited first, 30 of 36)

- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [Gemma 2: Improving Open Language Models at a Practical Size](https://mlsanity.com/paper/2408.00118/) (2024)
- [Understanding Black-box Predictions via Influence Functions](https://mlsanity.com/paper/1703.04730/) (2017)
- [ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness](https://mlsanity.com/paper/1811.12231/) (2018)
- [No Language Left Behind: Scaling Human-Centered Machine Translation](https://mlsanity.com/paper/2207.04672/) (2022)
- [Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference](https://mlsanity.com/paper/1902.01007/) (2019)
- [How multilingual is Multilingual BERT?](https://mlsanity.com/paper/1906.01502/) (2019)
- [Dissecting Recall of Factual Associations in Auto-Regressive Language Models](https://mlsanity.com/paper/2304.14767/) (2023)
- [Hallucination is Inevitable: An Innate Limitation of Large Language Models](https://mlsanity.com/paper/2401.11817/) (2024)
- [Data Distributional Properties Drive Emergent In-Context Learning in Transformers](https://mlsanity.com/paper/2205.05055/) (2022)
- [Patchscopes: A Unifying Framework for Inspecting Hidden Representations of Language Models](https://mlsanity.com/paper/2401.06102/) (2024)
- [A mathematical theory of semantic development in deep neural networks](https://mlsanity.com/paper/1810.10531/) (2018)
- [Cross-Lingual Ability of Multilingual BERT: An Empirical Study](https://mlsanity.com/paper/1912.07840/) (2019)
- [Emerging Cross-lingual Structure in Pretrained Language Models](https://mlsanity.com/paper/1911.01464/) (2019)
- [When Being Unseen from mBERT is just the Beginning: Handling New Languages With Multilingual Language Models](https://mlsanity.com/paper/2010.12858/) (2020)
- [What shapes feature representations? Exploring datasets, architectures, and training](https://mlsanity.com/paper/2006.12433/) (2020)
- [Cross-Lingual Consistency of Factual Knowledge in Multilingual Language Models](https://mlsanity.com/paper/2310.10378/) (2023)
- [Hopping Too Late: Exploring the Limitations of Large Language Models on Multi-Hop Queries](https://mlsanity.com/paper/2406.12775/) (2024)
- [Do Multilingual LLMs Think In English?](https://mlsanity.com/paper/2502.15603/) (2025)
- [First Align, then Predict: Understanding the Cross-Lingual Ability of Multilingual BERT](https://mlsanity.com/paper/2101.11109/) (2021)
- [On the Foundations of Shortcut Learning](https://mlsanity.com/paper/2310.16228/) (2023)
- [Competition Dynamics Shape Algorithmic Phases of In-Context Learning](https://mlsanity.com/paper/2412.01003/) (2024)
- [Lost in Multilinguality: Dissecting Cross-lingual Factual Inconsistency in Transformer Language Models](https://mlsanity.com/paper/2504.04264/) (2025)
- [Scalable Influence and Fact Tracing for Large Language Model Pretraining](https://mlsanity.com/paper/2410.17413/) (2024)
- [Procedural Knowledge in Pretraining Drives Reasoning in Large Language Models](https://mlsanity.com/paper/2411.12580/) (2024)
- [Theoretical and Practical Perspectives on what Influence Functions Do](https://mlsanity.com/paper/2305.16971/) (2023)
- [ECLeKTic: a Novel Challenge Set for Evaluation of Cross-Lingual Knowledge Transfer](https://mlsanity.com/paper/2502.21228/) (2025)
- [Overlap-based Vocabulary Generation Improves Cross-lingual Transfer Among Related Languages](https://mlsanity.com/paper/2203.01976/) (2022)
- [Learned feature representations are biased by complexity, learning order, position, and more](https://mlsanity.com/paper/2405.05847/) (2024)

## Built on by (most cited first, 3 of 3)

- [Rethinking Cross-lingual Alignment: Balancing Transfer and Cultural Erasure in Multilingual LLMs](https://mlsanity.com/paper/2510.26024/) (2025)
- [Rethinking Cross-lingual Gaps from a Statistical Viewpoint](https://mlsanity.com/paper/2510.15551/) (2025)
- [Adversarial Concept Search: Predicting Compositional Errors From Feature Geometry](https://mlsanity.com/paper/2606.13934/) (2026)

## Abstract (arXiv)

Large language models (LLMs) struggle with cross-lingual knowledge transfer: they sometimes hallucinate when asked in one language about facts expressed in a different language during training. This work introduces a controlled setting to study the causes and training dynamics of this phenomenon by training small Transformer models from scratch on synthetic multilingual datasets. Depending on (1) the correlation between facts and the language they were learned in (informativeness), and (2) the ease of language identification (extractability), models either develop unified representations across languages or separate representations; only when representations are unified do facts transfer across languages. Based on these insights, we propose a unifying perspective which explains a range of prior observations concerning cross-lingual transfer in multilingual LLMs. Our work shows controlled settings can shed light on pre-training dynamics and suggests methods to encourage representational unification as part of training that would improve LLMs' cross-lingual transfer.