# Robust Cross-lingual Embeddings from Parallel Sentences

arXiv: https://arxiv.org/abs/1912.12481 · HTML: https://mlsanity.com/paper/1912.12481/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 Dec 2019.

## Built on by (most cited first, 5 of 5)

- [Lightweight Cross-Lingual Sentence Representation Learning](https://mlsanity.com/paper/2105.13856/) (2021)
- [Cross-lingual Word Embeddings in Hyperbolic Space](https://mlsanity.com/paper/2205.01907/) (2022)
- [Leveraging Advantages of Interactive and Non-Interactive Models for Vector-Based Cross-Lingual Information Retrieval](https://mlsanity.com/paper/2111.01992/) (2021)
- [EMS: Efficient and Effective Massively Multilingual Sentence Embedding Learning](https://mlsanity.com/paper/2205.15744/) (2022)
- [Cross-Lingual Word Alignment for ASEAN Languages with Contrastive Learning](https://mlsanity.com/paper/2407.05054/) (2024)

## Abstract (arXiv)

Recent advances in cross-lingual word embeddings have primarily relied on mapping-based methods, which project pretrained word embeddings from different languages into a shared space through a linear transformation. However, these approaches assume word embedding spaces are isomorphic between different languages, which has been shown not to hold in practice (Sogaard et al., 2018), and fundamentally limits their performance. This motivates investigating joint learning methods which can overcome this impediment, by simultaneously learning embeddings across languages via a cross-lingual term in the training objective. We propose a bilingual extension of the CBOW method which leverages sentence-aligned corpora to obtain robust cross-lingual word and sentence representations. Our approach significantly improves cross-lingual sentence retrieval performance over all other approaches while maintaining parity with the current state-of-the-art methods on word-translation. It also achieves parity with a deep RNN method on a zero-shot cross-lingual document classification task, requiring far fewer computational resources for training and inference. As an additional advantage, our bilingual method leads to a much more pronounced improvement in the the quality of monolingual word vectors compared to other competing methods.