# A Survey Of Cross-lingual Word Embedding Models

arXiv: https://arxiv.org/abs/1706.04902 · HTML: https://mlsanity.com/paper/1706.04902/

## Key facts

- Cited by 143 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 15 Jun 2017.

## Built on by (most cited first, 30 of 143)

- [The Linear Representation Hypothesis and the Geometry of Large Language Models](https://mlsanity.com/paper/2311.03658/) (2023)
- [ChatGPT Beyond English: Towards a Comprehensive Evaluation of Large Language Models in Multilingual Learning](https://mlsanity.com/paper/2304.05613/) (2023)
- [Should ChatGPT be Biased? Challenges and Risks of Bias in Large Language Models](https://mlsanity.com/paper/2304.03738/) (2023)
- [It's All in the Heads: Using Attention Heads as a Baseline for Cross-Lingual Transfer in Commonsense Reasoning](https://mlsanity.com/paper/2106.12066/) (2021)
- [First Align, then Predict: Understanding the Cross-Lingual Ability of Multilingual BERT](https://mlsanity.com/paper/2101.11109/) (2021)
- [A Survey on Aspect-Based Sentiment Analysis: Tasks, Methods, and Challenges](https://mlsanity.com/paper/2203.01054/) (2022)
- [How to Adapt Your Pretrained Multilingual Model to 1600 Languages](https://mlsanity.com/paper/2106.02124/) (2021)
- [A Survey on Green Deep Learning](https://mlsanity.com/paper/2111.05193/) (2021)
- [A Primer on Pretrained Multilingual Language Models](https://mlsanity.com/paper/2107.00676/) (2021)
- [Revisiting the Primacy of English in Zero-shot Cross-lingual Transfer](https://mlsanity.com/paper/2106.16171/) (2021)
- [Expanding Pretrained Models to Thousands More Languages via Lexicon-based Adaptation](https://mlsanity.com/paper/2203.09435/) (2022)
- [Models and Datasets for Cross-Lingual Summarisation](https://mlsanity.com/paper/2202.09583/) (2022)
- [Enhancing Cross-lingual Transfer by Manifold Mixup](https://mlsanity.com/paper/2205.04182/) (2022)
- [On Cross-Lingual Retrieval with Multilingual Text Encoders](https://mlsanity.com/paper/2112.11031/) (2021)
- [Using Optimal Transport as Alignment Objective for fine-tuning Multilingual Contextualized Embeddings](https://mlsanity.com/paper/2110.02887/) (2021)
- [Hyperpolyglot LLMs: Cross-Lingual Interpretability in Token Embeddings](https://mlsanity.com/paper/2311.18034/) (2023)
- [Recent advances in text embedding: A Comprehensive Review of Top-Performing Methods on the MTEB Benchmark](https://mlsanity.com/paper/2406.01607/) (2024)
- [COVID-19 sentiment analysis via deep learning during the rise of novel cases](https://mlsanity.com/paper/2104.10662/) (2021)
- [XLM-K: Improving Cross-Lingual Language Model Pre-training with Multilingual Knowledge](https://mlsanity.com/paper/2109.12573/) (2021)
- [Improving Word Translation via Two-Stage Contrastive Learning](https://mlsanity.com/paper/2203.08307/) (2022)
- [Crossing the Conversational Chasm: A Primer on Natural Language Processing for Multilingual Task-Oriented Dialogue Systems](https://mlsanity.com/paper/2104.08570/) (2021)
- [Should we Stop Training More Monolingual Models, and Simply Use Machine Translation Instead?](https://mlsanity.com/paper/2104.10441/) (2021)
- [Cross-Lingual Ability of Multilingual Masked Language Models: A Study of Language Structure](https://mlsanity.com/paper/2203.08430/) (2022)
- [Abstractive Text Summarization: State of the Art, Challenges, and Improvements](https://mlsanity.com/paper/2409.02413/) (2024)
- [Evaluating Multilingual Text Encoders for Unsupervised Cross-Lingual Retrieval](https://mlsanity.com/paper/2101.08370/) (2021)
- [Modelling Latent Translations for Cross-Lingual Transfer](https://mlsanity.com/paper/2107.11353/) (2021)
- [Learning Backward Compatible Embeddings](https://mlsanity.com/paper/2206.03040/) (2022)
- [SLABERT Talk Pretty One Day: Modeling Second Language Acquisition with BERT](https://mlsanity.com/paper/2305.19589/) (2023)
- [Improving the Lexical Ability of Pretrained Language Models for Unsupervised Neural Machine Translation](https://mlsanity.com/paper/2103.10531/) (2021)
- [Cross-lingual Transfer of Monolingual Models](https://mlsanity.com/paper/2109.07348/) (2021)

## Abstract (arXiv)

Cross-lingual representations of words enable us to reason about word meaning in multilingual contexts and are a key facilitator of cross-lingual transfer when developing natural language processing models for low-resource languages. In this survey, we provide a comprehensive typology of cross-lingual word embedding models. We compare their data requirements and objective functions. The recurring theme of the survey is that many of the models presented in the literature optimize for the same objectives, and that seemingly different models are often equivalent modulo optimization strategies, hyper-parameters, and such. We also discuss the different ways cross-lingual word embeddings are evaluated, as well as future challenges and research horizons.