# Combining Static Word Embeddings and Contextual Representations for Bilingual Lexicon Induction

arXiv: https://arxiv.org/abs/2106.03084 · HTML: https://mlsanity.com/paper/2106.03084/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 43 references that are in the map.
- First posted on arXiv 6 Jun 2021.

## Builds on (most cited first, 23 of 23)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [Cross-lingual Language Model Pretraining](https://mlsanity.com/paper/1901.07291/) (2019)
- [Multilingual Denoising Pre-training for Neural Machine Translation](https://mlsanity.com/paper/2001.08210/) (2020)
- [Word Translation Without Parallel Data](https://mlsanity.com/paper/1710.04087/) (2017)
- [Exploiting Similarities among Languages for Machine Translation](https://mlsanity.com/paper/1309.4168/) (2013)
- [A robust self-learning method for fully unsupervised cross-lingual mappings of word embeddings](https://mlsanity.com/paper/1805.06297/) (2018)
- [A Survey Of Cross-lingual Word Embedding Models](https://mlsanity.com/paper/1706.04902/) (2017)
- [When and Why are Pre-trained Word Embeddings Useful for Neural Machine Translation?](https://mlsanity.com/paper/1804.06323/) (2018)
- [Multilingual Alignment of Contextual Word Representations](https://mlsanity.com/paper/2002.03518/) (2020)
- [Offline bilingual word vectors, orthogonal transformations and the inverted softmax](https://mlsanity.com/paper/1702.03859/) (2017)
- [Loss in Translation: Learning Bilingual Word Mapping with a Retrieval Criterion](https://mlsanity.com/paper/1804.07745/) (2018)
- [Cross-Lingual Alignment of Contextual Word Embeddings, with Applications to Zero-shot Dependency Parsing](https://mlsanity.com/paper/1902.09492/) (2019)
- [On the Limitations of Unsupervised Bilingual Dictionary Induction](https://mlsanity.com/paper/1805.03620/) (2018)
- [Improving zero-shot learning by mitigating the hubness problem](https://mlsanity.com/paper/1412.6568/) (2014)
- [Bilingual Lexicon Induction with Semi-supervision in Non-Isometric Embedding Spaces](https://mlsanity.com/paper/1908.06625/) (2019)
- [Cross-lingual Alignment vs Joint Training: A Comparative Study and A Simple Unified Framework](https://mlsanity.com/paper/1910.04708/) (2019)
- [Cross-lingual Alignment Methods for Multilingual BERT: A Comparative Study](https://mlsanity.com/paper/2009.14304/) (2020)
- [LNMap: Departures from Isomorphic Assumption in Bilingual Lexicon Induction Through Non-Linear Mapping in Latent Space](https://mlsanity.com/paper/2004.13889/) (2020)
- [Bilingual Lexicon Induction via Unsupervised Bitext Construction and Word Alignment](https://mlsanity.com/paper/2101.00148/) (2021)
- [Context-Aware Cross-Lingual Mapping](https://mlsanity.com/paper/1903.03243/) (2019)
- [Bilingual Lexicon Induction through Unsupervised Machine Translation](https://mlsanity.com/paper/1907.10761/) (2019)
- [Revisiting Adversarial Autoencoder for Unsupervised Word Translation with Cycle Consistency and Improved Training](https://mlsanity.com/paper/1904.04116/) (2019)

## Built on by (most cited first, 12 of 12)

- [A Survey on Multilingual Large Language Models: Corpora, Alignment, and Bias](https://mlsanity.com/paper/2404.00929/) (2024)
- [Hyperpolyglot LLMs: Cross-Lingual Interpretability in Token Embeddings](https://mlsanity.com/paper/2311.18034/) (2023)
- [Improving Word Translation via Two-Stage Contrastive Learning](https://mlsanity.com/paper/2203.08307/) (2022)
- [Improving Bilingual Lexicon Induction with Cross-Encoder Reranking](https://mlsanity.com/paper/2210.16953/) (2022)
- [When your Cousin has the Right Connections: Unsupervised Bilingual Lexicon Induction for Related Data-Imbalanced Languages](https://mlsanity.com/paper/2305.14012/) (2023)
- [On Bilingual Lexicon Induction with Large Language Models](https://mlsanity.com/paper/2310.13995/) (2023)
- [ProMap: Effective Bilingual Lexicon Induction via Language Model Prompting](https://mlsanity.com/paper/2310.18778/) (2023)
- [Leveraging Emotion-specific Features to Improve Transformer Performance for Emotion Classification](https://mlsanity.com/paper/2205.00283/) (2022)
- [A survey of neural-network-based methods utilising comparable data for finding translation equivalents](https://mlsanity.com/paper/2410.15144/) (2024)
- [Unsupervised Bilingual Lexicon Induction for Low Resource Languages](https://mlsanity.com/paper/2412.16894/) (2024)
- [Cross-Domain Bilingual Lexicon Induction via Pretrained Language Models](https://mlsanity.com/paper/2505.23146/) (2025)
- [How Good is BLI as an Alignment Measure: A Study in Word Embedding Paradigm](https://mlsanity.com/paper/2511.13040/) (2025)

## Abstract (arXiv)

Bilingual Lexicon Induction (BLI) aims to map words in one language to their translations in another, and is typically through learning linear projections to align monolingual word representation spaces. Two classes of word representations have been explored for BLI: static word embeddings and contextual representations, but there is no studies to combine both. In this paper, we propose a simple yet effective mechanism to combine the static word embeddings and the contextual representations to utilize the advantages of both paradigms. We test the combination mechanism on various language pairs under the supervised and unsupervised BLI benchmark settings. Experiments show that our mechanism consistently improves performances over robust BLI baselines on all language pairs by averagely improving 3.2 points in the supervised setting, and 3.1 points in the unsupervised setting.