mlsanityData to 7 Oct 2026

Improving Word Translation via Two-Stage Contrastive Learning

arXiv 2203.08307 ↗First posted 15 Mar 2022cs.CL cs.AI cs.IRcited by 18

Not on the map. Why?

Abstract (arXiv)

Word translation or bilingual lexicon induction (BLI) is a key cross-lingual task, aiming to bridge the lexical gap between different languages. In this work, we propose a robust and effective two-stage contrastive learning framework for the BLI task. At Stage C1, we propose to refine standard cross-lingual linear maps between static word embeddings (WEs) via a contrastive learning objective; we also show how to integrate it into the self-learning procedure for even more refined cross-lingual maps. In Stage C2, we conduct BLI-oriented contrastive fine-tuning of mBERT, unlocking its word translation capability. We also show that static WEs induced from the `C2-tuned' mBERT complement static WEs from Stage C1. Comprehensive experiments on standard BLI datasets for diverse languages and different experimental setups demonstrate substantial gains achieved by our framework. While the BLI method from Stage C1 already yields substantial gains over all state-of-the-art BLI methods in our comparison, even stronger improvements are met with the full two-stage framework: e.g., we report gains for 112/112 BLI setups, spanning 28 language pairs.

Key facts. Cited by 18 papers in the map (data through 7 Oct 2026). Builds on 44 of its 69 references that are in the map. First posted on arXiv 15 Mar 2022.

Builds on

44 of 69 references.

References · 44

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  3. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  4. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,168
  5. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,813
  6. 2016Enriching Word Vectors with Subword Informationcited by 1,578
  7. 2019Cross-lingual Language Model Pretrainingcited by 1,088
  8. 2017Word Translation Without Parallel Datacited by 480
  9. 2020A Metric Learning Reality Checkcited by 317
  10. 2020DeCLUTR: Deep Contrastive Learning for Unsupervised Textual Representationscited by 249
  11. 2020Self-Alignment Pretraining for Biomedical Entity Representationscited by 233
  12. 2013Exploiting Similarities among Languages for Machine Translationcited by 187
32 more
  1. 2020Probing Pretrained Language Models for Lexical Semanticscited by 156
  2. 2018A robust self-learning method for fully unsupervised cross-lingual mappings of word embeddingscited by 145
  3. 2021Composable Sparse Fine-Tuning for Cross-Lingual Transfercited by 144
  4. 2017A Survey Of Cross-lingual Word Embedding Modelscited by 142
  5. 2021SimCLS: A Simple Framework for Contrastive Learning of Abstractive Summarizationcited by 131
  6. 2018When and Why are Pre-trained Word Embeddings Useful for Neural Machine Translation?cited by 125
  7. 2021Fast, Effective, and Self-Supervised: Transforming Masked Language Models into Universal Lexical and Sentence Encoderscited by 88
  8. 2020On the Language Neutrality of Pre-trained Multilingual Representationscited by 82
  9. 2018Loss in Translation: Learning Bilingual Word Mapping with a Retrieval Criterioncited by 69
  10. 2021CONTaiNER: Few-Shot Named Entity Recognition via Contrastive Learningcited by 64
  11. 2018On the Limitations of Unsupervised Bilingual Dictionary Inductioncited by 61
  12. 2019How to (Properly) Evaluate Cross-Lingual Word Embeddings: On Strong Baselines, Comparative Analyses, and Some Misconceptionscited by 48
  13. 2021Phrase-BERT: Improved Phrase Embeddings from BERT with an Application to Corpus Explorationcited by 44
  14. 2020Multi-SimLex: A Large-Scale Evaluation of Multilingual and Cross-Lingual Lexical Semantic Similaritycited by 38
  15. 2020A Call for More Rigor in Unsupervised Cross-lingual Learningcited by 37
  16. 2019Bilingual Lexicon Induction with Semi-supervision in Non-Isometric Embedding Spacescited by 35
  17. 2019Do We Really Need Fully Unsupervised Cross-Lingual Embeddings?cited by 31
  18. 2020It's not Greek to mBERT: Inducing Word-Level Translations from Multilingual BERTcited by 30
  19. 2021Lexicon Learning for Few-Shot Neural Sequence Modelingcited by 28
  20. 2020LNMap: Departures from Isomorphic Assumption in Bilingual Lexicon Induction Through Non-Linear Mapping in Latent Spacecited by 23
  21. 2018Learning Multilingual Word Embeddings in Latent Metric Space: A Geometric Approachcited by 22
  22. 2021Specializing Multilingual Language Models: An Empirical Studycited by 21
  23. 2018Non-Adversarial Unsupervised Word Translationcited by 19
  24. 2020Bilingual Dictionary Based Neural Machine Translation without Using Parallel Sentencescited by 19
  25. 2021Bilingual Lexicon Induction via Unsupervised Bitext Construction and Word Alignmentcited by 19
  26. 2021MirrorWiC: On Eliciting Word-in-Context Representations from Pretrained Language Modelscited by 18
  27. 2021Improving the Lexical Ability of Pretrained Language Models for Unsupervised Neural Machine Translationcited by 14
  28. 2019Interactive Refinement of Cross-Lingual Word Embeddingscited by 13
  29. 2019Revisiting Adversarial Autoencoder for Unsupervised Word Translation with Cycle Consistency and Improved Trainingcited by 12
  30. 2021Combining Static Word Embeddings and Contextual Representations for Bilingual Lexicon Inductioncited by 12
  31. 2020Plan Optimization to Bilingual Dictionary Induction for Low-Resource Language Familiescited by 3
  32. 2021Learning a Reversible Embedding Mapping using Bi-Directional Manifold Alignmentcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 3; 2023: 6; 2024: 6; 2025: 1; 2026: 2.

18 papers.

Sort:

  1. 2024A Survey on Multilingual Large Language Models: Corpora, Alignment, and Biascited by 82
  2. 2024Heterogeneous Contrastive Learning for Foundation Models and Beyondcited by 27
  3. 2024CALRec: Contrastive Alignment of Generative LLMs for Sequential Recommendationcited by 23
  4. 2023Translation-Enhanced Multilingual Text-to-Image Generationcited by 9
  5. 2022Probing Cross-Lingual Lexical Knowledge from Multilingual Sentence Encoderscited by 7
  6. 2022IsoVec: Controlling the Relative Isomorphism of Word Embedding Spacescited by 7
  7. 2022Improving Bilingual Lexicon Induction with Cross-Encoder Rerankingcited by 7
  8. 2023Medical Question Summarization with Entity-driven Contrastive Learningcited by 5
  9. 2023Mitigating Data Imbalance and Representation Degeneration in Multilingual Machine Translationcited by 5
  10. 2024Self-Augmented In-Context Learning for Unsupervised Word Translationcited by 4
  11. 2023Translate to Disambiguate: Zero-shot Multilingual Word Sense Disambiguation with Pretrained Language Modelscited by 2
  12. 2023On Bilingual Lexicon Induction with Large Language Modelscited by 2
6 more
  1. 2023ProMap: Effective Bilingual Lexicon Induction via Language Model Promptingcited by 2
  2. 2026TokAlign++: Advancing Vocabulary Adaptation via Better Token Alignmentcited by 1
  3. 2024How Lexical is Bilingual Lexicon Induction?cited by 0
  4. 2024A survey of neural-network-based methods utilising comparable data for finding translation equivalentscited by 0
  5. 2025How Good is BLI as an Alignment Measure: A Study in Word Embedding Paradigmcited by 0
  6. 2026ViCLSR: A Supervised Contrastive Learning Framework with Natural Language Inference for Natural Language Understanding Taskscited by 0