# Isomorphic Cross-lingual Embeddings for Low-Resource Languages

arXiv: https://arxiv.org/abs/2203.14632 · HTML: https://mlsanity.com/paper/2203.14632/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 44 references that are in the map.
- First posted on arXiv 28 Mar 2022.

## Builds on (most cited first, 29 of 29)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [On the Properties of Neural Machine Translation: Encoder-Decoder Approaches](https://mlsanity.com/paper/1409.1259/) (2014)
- [Cross-lingual Language Model Pretraining](https://mlsanity.com/paper/1901.07291/) (2019)
- [Word Translation Without Parallel Data](https://mlsanity.com/paper/1710.04087/) (2017)
- [Learning Word Vectors for 157 Languages](https://mlsanity.com/paper/1802.06893/) (2018)
- [Analysis Methods in Neural Language Processing: A Survey](https://mlsanity.com/paper/1812.08951/) (2018)
- [Unsupervised Machine Translation Using Monolingual Corpora Only](https://mlsanity.com/paper/1711.00043/) (2017)
- [Exploiting Similarities among Languages for Machine Translation](https://mlsanity.com/paper/1309.4168/) (2013)
- [Unsupervised Neural Machine Translation](https://mlsanity.com/paper/1710.11041/) (2017)
- [A robust self-learning method for fully unsupervised cross-lingual mappings of word embeddings](https://mlsanity.com/paper/1805.06297/) (2018)
- [A Survey Of Cross-lingual Word Embedding Models](https://mlsanity.com/paper/1706.04902/) (2017)
- [Phrase-Based & Neural Unsupervised Machine Translation](https://mlsanity.com/paper/1804.07755/) (2018)
- [Achieving Human Parity on Automatic Chinese to English News Translation](https://mlsanity.com/paper/1803.05567/) (2018)
- [The FLoRes Evaluation Datasets for Low-Resource Machine Translation: Nepali-English and Sinhala-English](https://mlsanity.com/paper/1902.01382/) (2019)
- [On the Limitations of Unsupervised Bilingual Dictionary Induction](https://mlsanity.com/paper/1805.03620/) (2018)
- [Very Deep Transformers for Neural Machine Translation](https://mlsanity.com/paper/2008.07772/) (2020)
- [Are All Good Word Vector Spaces Isomorphic?](https://mlsanity.com/paper/2004.04070/) (2020)
- [Bilingual Lexicon Induction with Semi-supervision in Non-Isometric Embedding Spaces](https://mlsanity.com/paper/1908.06625/) (2019)
- [Do We Really Need Fully Unsupervised Cross-Lingual Embeddings?](https://mlsanity.com/paper/1909.01638/) (2019)
- [Cross-lingual Alignment vs Joint Training: A Comparative Study and A Simple Unified Framework](https://mlsanity.com/paper/1910.04708/) (2019)
- [LNMap: Departures from Isomorphic Assumption in Bilingual Lexicon Induction Through Non-Linear Mapping in Latent Space](https://mlsanity.com/paper/2004.13889/) (2020)
- [Analyzing the Limitations of Cross-lingual Word Embedding Mappings](https://mlsanity.com/paper/1906.05407/) (2019)
- [BilBOWA: Fast Bilingual Distributed Representations without Word Alignments](https://mlsanity.com/paper/1410.2455/) (2014)
- [Learning Crosslingual Word Embeddings without Bilingual Corpora](https://mlsanity.com/paper/1606.09403/) (2016)
- [Beyond Offline Mapping: Learning Cross Lingual Word Embeddings through Context Anchoring](https://mlsanity.com/paper/2012.15715/) (2020)
- [Bilingual Distributed Word Representations from Document-Aligned Comparable Data](https://mlsanity.com/paper/1509.07308/) (2015)
- [Summarising Historical Text in Modern Languages](https://mlsanity.com/paper/2101.10759/) (2021)

## Built on by (most cited first, 1 of 1)

- [A survey of neural-network-based methods utilising comparable data for finding translation equivalents](https://mlsanity.com/paper/2410.15144/) (2024)

## Abstract (arXiv)

Cross-Lingual Word Embeddings (CLWEs) are a key component to transfer linguistic information learnt from higher-resource settings into lower-resource ones. Recent research in cross-lingual representation learning has focused on offline mapping approaches due to their simplicity, computational efficacy, and ability to work with minimal parallel resources. However, they crucially depend on the assumption of embedding spaces being approximately isomorphic i.e. sharing similar geometric structure, which does not hold in practice, leading to poorer performance on low-resource and distant language pairs. In this paper, we introduce a framework to learn CLWEs, without assuming isometry, for low-resource pairs via joint exploitation of a related higher-resource language. In our work, we first pre-align the low-resource and related language embedding spaces using offline methods to mitigate the assumption of isometry. Following this, we use joint training methods to develops CLWEs for the related language and the target embed-ding space. Finally, we remap the pre-aligned low-resource space and the target space to generate the final CLWEs. We show consistent gains over current methods in both quality and degree of isomorphism, as measured by bilingual lexicon induction (BLI) and eigenvalue similarity respectively, across several language pairs: {Nepali, Finnish, Romanian, Gujarati, Hungarian}-English. Lastly, our analysis also points to the relatedness as well as the amount of related language data available as being key factors in determining the quality of embeddings achieved.