# Improving Word Translation via Two-Stage Contrastive Learning

arXiv: https://arxiv.org/abs/2203.08307 · HTML: https://mlsanity.com/paper/2203.08307/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Builds on 44 of its 69 references that are in the map.
- First posted on arXiv 15 Mar 2022.

## Builds on (most cited first, 30 of 44)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [SimCSE: Simple Contrastive Learning of Sentence Embeddings](https://mlsanity.com/paper/2104.08821/) (2021)
- [mT5: A massively multilingual pre-trained text-to-text transformer](https://mlsanity.com/paper/2010.11934/) (2020)
- [Enriching Word Vectors with Subword Information](https://mlsanity.com/paper/1607.04606/) (2016)
- [Cross-lingual Language Model Pretraining](https://mlsanity.com/paper/1901.07291/) (2019)
- [Word Translation Without Parallel Data](https://mlsanity.com/paper/1710.04087/) (2017)
- [A Metric Learning Reality Check](https://mlsanity.com/paper/2003.08505/) (2020)
- [DeCLUTR: Deep Contrastive Learning for Unsupervised Textual Representations](https://mlsanity.com/paper/2006.03659/) (2020)
- [Self-Alignment Pretraining for Biomedical Entity Representations](https://mlsanity.com/paper/2010.11784/) (2020)
- [Exploiting Similarities among Languages for Machine Translation](https://mlsanity.com/paper/1309.4168/) (2013)
- [Probing Pretrained Language Models for Lexical Semantics](https://mlsanity.com/paper/2010.05731/) (2020)
- [A robust self-learning method for fully unsupervised cross-lingual mappings of word embeddings](https://mlsanity.com/paper/1805.06297/) (2018)
- [Composable Sparse Fine-Tuning for Cross-Lingual Transfer](https://mlsanity.com/paper/2110.07560/) (2021)
- [A Survey Of Cross-lingual Word Embedding Models](https://mlsanity.com/paper/1706.04902/) (2017)
- [SimCLS: A Simple Framework for Contrastive Learning of Abstractive Summarization](https://mlsanity.com/paper/2106.01890/) (2021)
- [When and Why are Pre-trained Word Embeddings Useful for Neural Machine Translation?](https://mlsanity.com/paper/1804.06323/) (2018)
- [Fast, Effective, and Self-Supervised: Transforming Masked Language Models into Universal Lexical and Sentence Encoders](https://mlsanity.com/paper/2104.08027/) (2021)
- [On the Language Neutrality of Pre-trained Multilingual Representations](https://mlsanity.com/paper/2004.05160/) (2020)
- [Loss in Translation: Learning Bilingual Word Mapping with a Retrieval Criterion](https://mlsanity.com/paper/1804.07745/) (2018)
- [CONTaiNER: Few-Shot Named Entity Recognition via Contrastive Learning](https://mlsanity.com/paper/2109.07589/) (2021)
- [On the Limitations of Unsupervised Bilingual Dictionary Induction](https://mlsanity.com/paper/1805.03620/) (2018)
- [How to (Properly) Evaluate Cross-Lingual Word Embeddings: On Strong Baselines, Comparative Analyses, and Some Misconceptions](https://mlsanity.com/paper/1902.00508/) (2019)
- [Phrase-BERT: Improved Phrase Embeddings from BERT with an Application to Corpus Exploration](https://mlsanity.com/paper/2109.06304/) (2021)
- [Multi-SimLex: A Large-Scale Evaluation of Multilingual and Cross-Lingual Lexical Semantic Similarity](https://mlsanity.com/paper/2003.04866/) (2020)
- [A Call for More Rigor in Unsupervised Cross-lingual Learning](https://mlsanity.com/paper/2004.14958/) (2020)
- [Bilingual Lexicon Induction with Semi-supervision in Non-Isometric Embedding Spaces](https://mlsanity.com/paper/1908.06625/) (2019)
- [Do We Really Need Fully Unsupervised Cross-Lingual Embeddings?](https://mlsanity.com/paper/1909.01638/) (2019)
- [It's not Greek to mBERT: Inducing Word-Level Translations from Multilingual BERT](https://mlsanity.com/paper/2010.08275/) (2020)

## Built on by (most cited first, 18 of 18)

- [A Survey on Multilingual Large Language Models: Corpora, Alignment, and Bias](https://mlsanity.com/paper/2404.00929/) (2024)
- [Heterogeneous Contrastive Learning for Foundation Models and Beyond](https://mlsanity.com/paper/2404.00225/) (2024)
- [CALRec: Contrastive Alignment of Generative LLMs for Sequential Recommendation](https://mlsanity.com/paper/2405.02429/) (2024)
- [Translation-Enhanced Multilingual Text-to-Image Generation](https://mlsanity.com/paper/2305.19216/) (2023)
- [Probing Cross-Lingual Lexical Knowledge from Multilingual Sentence Encoders](https://mlsanity.com/paper/2205.00267/) (2022)
- [IsoVec: Controlling the Relative Isomorphism of Word Embedding Spaces](https://mlsanity.com/paper/2210.05098/) (2022)
- [Improving Bilingual Lexicon Induction with Cross-Encoder Reranking](https://mlsanity.com/paper/2210.16953/) (2022)
- [Medical Question Summarization with Entity-driven Contrastive Learning](https://mlsanity.com/paper/2304.07437/) (2023)
- [Mitigating Data Imbalance and Representation Degeneration in Multilingual Machine Translation](https://mlsanity.com/paper/2305.12786/) (2023)
- [Self-Augmented In-Context Learning for Unsupervised Word Translation](https://mlsanity.com/paper/2402.10024/) (2024)
- [Translate to Disambiguate: Zero-shot Multilingual Word Sense Disambiguation with Pretrained Language Models](https://mlsanity.com/paper/2304.13803/) (2023)
- [On Bilingual Lexicon Induction with Large Language Models](https://mlsanity.com/paper/2310.13995/) (2023)
- [ProMap: Effective Bilingual Lexicon Induction via Language Model Prompting](https://mlsanity.com/paper/2310.18778/) (2023)
- [TokAlign++: Advancing Vocabulary Adaptation via Better Token Alignment](https://mlsanity.com/paper/2605.13429/) (2026)
- [How Lexical is Bilingual Lexicon Induction?](https://mlsanity.com/paper/2404.04221/) (2024)
- [A survey of neural-network-based methods utilising comparable data for finding translation equivalents](https://mlsanity.com/paper/2410.15144/) (2024)
- [How Good is BLI as an Alignment Measure: A Study in Word Embedding Paradigm](https://mlsanity.com/paper/2511.13040/) (2025)
- [ViCLSR: A Supervised Contrastive Learning Framework with Natural Language Inference for Natural Language Understanding Tasks](https://mlsanity.com/paper/2603.21084/) (2026)

## Abstract (arXiv)

Word translation or bilingual lexicon induction (BLI) is a key cross-lingual task, aiming to bridge the lexical gap between different languages. In this work, we propose a robust and effective two-stage contrastive learning framework for the BLI task. At Stage C1, we propose to refine standard cross-lingual linear maps between static word embeddings (WEs) via a contrastive learning objective; we also show how to integrate it into the self-learning procedure for even more refined cross-lingual maps. In Stage C2, we conduct BLI-oriented contrastive fine-tuning of mBERT, unlocking its word translation capability. We also show that static WEs induced from the `C2-tuned' mBERT complement static WEs from Stage C1. Comprehensive experiments on standard BLI datasets for diverse languages and different experimental setups demonstrate substantial gains achieved by our framework. While the BLI method from Stage C1 already yields substantial gains over all state-of-the-art BLI methods in our comparison, even stronger improvements are met with the full two-stage framework: e.g., we report gains for 112/112 BLI setups, spanning 28 language pairs.