# Cross-lingual Similarity of Multilingual Representations Revisited

arXiv: https://arxiv.org/abs/2212.01924 · HTML: https://mlsanity.com/paper/2212.01924/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 23 references that are in the map.
- First posted on arXiv 4 Dec 2022.

## Builds on (most cited first, 17 of 17)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Similarity of Neural Network Representations Revisited](https://mlsanity.com/paper/1905.00414/) (2019)
- [On Layer Normalization in the Transformer Architecture](https://mlsanity.com/paper/2002.04745/) (2020)
- [XNLI: Evaluating Cross-lingual Sentence Representations](https://mlsanity.com/paper/1809.05053/) (2018)
- [How multilingual is Multilingual BERT?](https://mlsanity.com/paper/1906.01502/) (2019)
- [Insights on representational similarity in neural networks with canonical correlation](https://mlsanity.com/paper/1806.05759/) (2018)
- [Beto, Bentz, Becas: The Surprising Cross-Lingual Effectiveness of BERT](https://mlsanity.com/paper/1904.09077/) (2019)
- [Convergent Learning: Do different neural networks learn the same representations?](https://mlsanity.com/paper/1511.07543/) (2015)
- [Emerging Cross-lingual Structure in Pretrained Language Models](https://mlsanity.com/paper/1911.01464/) (2019)
- [On the Language Neutrality of Pre-trained Multilingual Representations](https://mlsanity.com/paper/2004.05160/) (2020)
- [NormFormer: Improved Transformer Pretraining with Extra Normalization](https://mlsanity.com/paper/2110.09456/) (2021)
- [Investigating Multilingual NMT Representations at Scale](https://mlsanity.com/paper/1909.02197/) (2019)
- [First Align, then Predict: Understanding the Cross-Lingual Ability of Multilingual BERT](https://mlsanity.com/paper/2101.11109/) (2021)
- [The Dangers of Underclaiming: Reasons for Caution When Reporting How NLP Systems Fail](https://mlsanity.com/paper/2110.08300/) (2021)
- [Multilingual Graphemic Hybrid ASR with Massive Data Augmentation](https://mlsanity.com/paper/1909.06522/) (2019)
- [Similarity of Sentence Representations in Multilingual LMs: Resolving Conflicting Literature and Case Study of Baltic Languages](https://mlsanity.com/paper/2109.01207/) (2021)

## Built on by (most cited first, 8 of 8)

- [The Same But Different: Structural Similarities and Differences in Multilingual Language Modeling](https://mlsanity.com/paper/2410.09223/) (2024)
- [Representation Potentials of Foundation Models for Multimodal Alignment: A Survey](https://mlsanity.com/paper/2510.05184/) (2025)
- [From Neurons to Semantics: Evaluating Cross-Linguistic Alignment Capabilities of Large Language Models via Neurons Alignment](https://mlsanity.com/paper/2507.14900/) (2025)
- [Can Embedding Similarity Predict Cross-Lingual Transfer? A Systematic Study on African Languages](https://mlsanity.com/paper/2601.03168/) (2026)
- [High-Dimensional Interlingual Representations of Large Language Models](https://mlsanity.com/paper/2503.11280/) (2025)
- [Few-Shot Spoken Language Understanding via Joint Speech-Text Models](https://mlsanity.com/paper/2310.05919/) (2023)
- [Predicting Multilingual Classification and Translation Performance of LLMs with Cross-Lingual Alignment -- Is English Enough?](https://mlsanity.com/paper/2608.03446/) (2026)
- [A Systematic Comparison of Multilingual Interpretability Methods Reveals Anisotropy-Driven Failures](https://mlsanity.com/paper/2609.04819/) (2026)

## Abstract (arXiv)

Related works used indexes like CKA and variants of CCA to measure the similarity of cross-lingual representations in multilingual language models. In this paper, we argue that assumptions of CKA/CCA align poorly with one of the motivating goals of cross-lingual learning analysis, i.e., explaining zero-shot cross-lingual transfer. We highlight what valuable aspects of cross-lingual similarity these indexes fail to capture and provide a motivating case study textitdemonstrating the problem empirically. Then, we introduce textitAverage Neuron-Wise Correlation (ANC) as a straightforward alternative that is exempt from the difficulties of CKA/CCA and is good specifically in a cross-lingual context. Finally, we use ANC to construct evidence that the previously introduced ``first align, then predict'' pattern takes place not only in masked language models (MLMs) but also in multilingual models with textitcausal language modeling objectives (CLMs). Moreover, we show that the pattern extends to the textitscaled versions of the MLMs and CLMs (up to 85x original mBERT).footnoteOur code is publicly available at urlhttps://github.com/TartuNLP/xsim