# A Massively Multilingual Analysis of Cross-linguality in Shared Embedding Space arXiv: https://arxiv.org/abs/2109.06324 ยท HTML: https://mlsanity.com/paper/2109.06324/ ## Key facts - Cited by 8 papers in the map (data through 14 Oct 2026). - Builds on 35 of its 57 references that are in the map. - First posted on arXiv 13 Sep 2021. ## Builds on (most cited first, 30 of 35) - [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013) - [Billion-scale similarity search with GPUs](https://mlsanity.com/paper/1702.08734/) (2017) - [Enriching Word Vectors with Subword Information](https://mlsanity.com/paper/1607.04606/) (2016) - [Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation](https://mlsanity.com/paper/1609.08144/) (2016) - [Energy and Policy Considerations for Deep Learning in NLP](https://mlsanity.com/paper/1906.02243/) (2019) - [Bag of Tricks for Efficient Text Classification](https://mlsanity.com/paper/1607.01759/) (2016) - [A Primer in BERTology: What we know about how BERT works](https://mlsanity.com/paper/2002.12327/) (2020) - [The State and Fate of Linguistic Diversity and Inclusion in the NLP World](https://mlsanity.com/paper/2004.09095/) (2020) - [How multilingual is Multilingual BERT?](https://mlsanity.com/paper/1906.01502/) (2019) - [Green AI](https://mlsanity.com/paper/1907.10597/) (2019) - [Language-agnostic BERT Sentence Embedding](https://mlsanity.com/paper/2007.01852/) (2020) - [Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillation](https://mlsanity.com/paper/2004.09813/) (2020) - [Beyond English-Centric Multilingual Machine Translation](https://mlsanity.com/paper/2010.11125/) (2020) - [Google's Multilingual Neural Machine Translation System: Enabling Zero-Shot Translation](https://mlsanity.com/paper/1611.04558/) (2016) - [Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyond](https://mlsanity.com/paper/1812.10464/) (2018) - [Beto, Bentz, Becas: The Surprising Cross-Lingual Effectiveness of BERT](https://mlsanity.com/paper/1904.09077/) (2019) - [WikiMatrix: Mining 135M Parallel Sentences in 1620 Language Pairs from Wikipedia](https://mlsanity.com/paper/1907.05791/) (2019) - [Are All Languages Created Equal in Multilingual BERT?](https://mlsanity.com/paper/2005.09093/) (2020) - [Cross-Lingual Ability of Multilingual BERT: An Empirical Study](https://mlsanity.com/paper/1912.07840/) (2019) - [Emerging Cross-lingual Structure in Pretrained Language Models](https://mlsanity.com/paper/1911.01464/) (2019) - [CCMatrix: Mining Billions of High-Quality Parallel Sentences on the WEB](https://mlsanity.com/paper/1911.04944/) (2019) - [Choosing Transfer Languages for Cross-Lingual Learning](https://mlsanity.com/paper/1905.12688/) (2019) - [The Tatoeba Translation Challenge -- Realistic Data Sets for Low Resource and Multilingual MT](https://mlsanity.com/paper/2010.06354/) (2020) - [Margin-based Parallel Corpus Mining with Multilingual Sentence Embeddings](https://mlsanity.com/paper/1811.01136/) (2018) - [Cross-lingual Retrieval for Iterative Self-Supervised Training](https://mlsanity.com/paper/2006.09526/) (2020) - [Learning Language Representations for Typology Prediction](https://mlsanity.com/paper/1707.09569/) (2017) - [Are All Good Word Vector Spaces Isomorphic?](https://mlsanity.com/paper/2004.04070/) (2020) - [Challenges of language technologies for the indigenous languages of the Americas](https://mlsanity.com/paper/1806.04291/) (2018) - [MetaXL: Meta Representation Transformation for Low-resource Cross-lingual Learning](https://mlsanity.com/paper/2104.07908/) (2021) - [Towards Realistic Practices In Low-Resource Natural Language Processing: The Development Set](https://mlsanity.com/paper/1909.01522/) (2019) ## Built on by (most cited first, 8 of 8) - [Goldfish: Monolingual Language Models for 350 Languages](https://mlsanity.com/paper/2408.10441/) (2024) - [Structural Priming Demonstrates Abstract Grammatical Representations in Multilingual Language Models](https://mlsanity.com/paper/2311.09194/) (2023) - [Beyond Literal Token Overlap: Token Alignability for Multilinguality](https://mlsanity.com/paper/2502.06468/) (2025) - [On the Calibration of Multilingual Question Answering LLMs](https://mlsanity.com/paper/2311.08669/) (2023) - [AugCSE: Contrastive Sentence Embedding with Diverse Augmentations](https://mlsanity.com/paper/2210.13749/) (2022) - [Uncovering inequalities in new knowledge learning by large language models across different languages](https://mlsanity.com/paper/2503.04064/) (2025) - [Finetuning a Kalaallisut-English machine translation system using web-crawled data](https://mlsanity.com/paper/2206.02230/) (2022) - [HydraQE: OSU's Submission for the IWSLT 2026 Speech Translation Metrics Shared Task](https://mlsanity.com/paper/2606.08748/) (2026) ## Abstract (arXiv) In cross-lingual language models, representations for many different languages live in the same space. Here, we investigate the linguistic and non-linguistic factors affecting sentence-level alignment in cross-lingual pretrained language models for 101 languages and 5,050 language pairs. Using BERT-based LaBSE and BiLSTM-based LASER as our models, and the Bible as our corpus, we compute a task-based measure of cross-lingual alignment in the form of bitext retrieval performance, as well as four intrinsic measures of vector space alignment and isomorphism. We then examine a range of linguistic, quasi-linguistic, and training-related features as potential predictors of these alignment metrics. The results of our analyses show that word order agreement and agreement in morphological complexity are two of the strongest linguistic predictors of cross-linguality. We also note in-family training data as a stronger predictor than language-specific training data across the board. We verify some of our linguistic findings by looking at the effect of morphological segmentation on English-Inuktitut alignment, in addition to examining the effect of word order agreement on isomorphism for 66 zero-shot language pairs from a different corpus. We make the data and code for our experiments publicly available.