# Cross-Lingual BERT Contextual Embedding Space Mapping with Isotropic and Isometric Conditions

arXiv: https://arxiv.org/abs/2107.09186 · HTML: https://mlsanity.com/paper/2107.09186/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 30 references that are in the map.
- First posted on arXiv 19 Jul 2021.

## Builds on (most cited first, 16 of 16)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://mlsanity.com/paper/1906.08237/) (2019)
- [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018)
- [Enriching Word Vectors with Subword Information](https://mlsanity.com/paper/1607.04606/) (2016)
- [How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddings](https://mlsanity.com/paper/1909.00512/) (2019)
- [Word Translation Without Parallel Data](https://mlsanity.com/paper/1710.04087/) (2017)
- [Exploiting Similarities among Languages for Machine Translation](https://mlsanity.com/paper/1309.4168/) (2013)
- [A Survey Of Cross-lingual Word Embedding Models](https://mlsanity.com/paper/1706.04902/) (2017)
- [KUISAIL at SemEval-2020 Task 12: BERT-CNN for Offensive Speech Identification in Social Media](https://mlsanity.com/paper/2007.13184/) (2020)
- [Cross-Lingual Alignment of Contextual Word Embeddings, with Applications to Zero-shot Dependency Parsing](https://mlsanity.com/paper/1902.09492/) (2019)
- [On the Limitations of Unsupervised Bilingual Dictionary Induction](https://mlsanity.com/paper/1805.03620/) (2018)
- [Are All Good Word Vector Spaces Isomorphic?](https://mlsanity.com/paper/2004.04070/) (2020)
- [BilBOWA: Fast Bilingual Distributed Representations without Word Alignments](https://mlsanity.com/paper/1410.2455/) (2014)
- [Are Girls Neko or Sh=ojo? Cross-Lingual Alignment of Non-Isomorphic Embeddings with Iterative Normalization](https://mlsanity.com/paper/1906.01622/) (2019)
- [Towards cross-lingual distributed representations without parallel text trained with adversarial autoencoders](https://mlsanity.com/paper/1608.02996/) (2016)

## Built on by (most cited first, 6 of 6)

- [Middle-Layer Representation Alignment for Cross-Lingual Transfer in Fine-Tuned LLMs](https://mlsanity.com/paper/2502.14830/) (2025)
- [An Isotropy Analysis in the Multilingual BERT Embedding Space](https://mlsanity.com/paper/2110.04504/) (2021)
- [Voices in a Crowd: Searching for Clusters of Unique Perspectives](https://mlsanity.com/paper/2407.14259/) (2024)
- [Por Qu'e N~ao Utiliser Alla Spraak? Mixed Training with Gradient Optimization in Few-Shot Cross-Lingual Transfer](https://mlsanity.com/paper/2204.13869/) (2022)
- [A survey of neural-network-based methods utilising comparable data for finding translation equivalents](https://mlsanity.com/paper/2410.15144/) (2024)
- [Vec2Summ: Text Summarization via Probabilistic Sentence Embeddings](https://mlsanity.com/paper/2508.07017/) (2025)

## Abstract (arXiv)

Typically, a linearly orthogonal transformation mapping is learned by aligning static type-level embeddings to build a shared semantic space. In view of the analysis that contextual embeddings contain richer semantic features, we investigate a context-aware and dictionary-free mapping approach by leveraging parallel corpora. We illustrate that our contextual embedding space mapping significantly outperforms previous multilingual word embedding methods on the bilingual dictionary induction (BDI) task by providing a higher degree of isomorphism. To improve the quality of mapping, we also explore sense-level embeddings that are split from type-level representations, which can align spaces in a finer resolution and yield more precise mapping. Moreover, we reveal that contextual embedding spaces suffer from their natural properties -- anisotropy and anisometry. To mitigate these two problems, we introduce the iterative normalization algorithm as an imperative preprocessing step. Our findings unfold the tight relationship between isotropy, isometry, and isomorphism in normalized contextual embedding spaces.