# Why Overfitting Isn't Always Bad: Retrofitting Cross-Lingual Word Embeddings to Dictionaries

arXiv: https://arxiv.org/abs/2005.00524 · HTML: https://mlsanity.com/paper/2005.00524/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 1 May 2020.

## Built on by (most cited first, 7 of 7)

- [State-of-the-art generalisation research in NLP: A taxonomy and review](https://mlsanity.com/paper/2210.03050/) (2022)
- [A Dataset and Baselines for Multilingual Reply Suggestion](https://mlsanity.com/paper/2106.02017/) (2021)
- [Extracting Sentence Embeddings from Pretrained Transformer Models](https://mlsanity.com/paper/2408.08073/) (2024)
- [Bilingual Lexicon Induction for Low-Resource Languages using Graph Matching via Optimal Transport](https://mlsanity.com/paper/2210.14378/) (2022)
- [An Analysis of Euclidean vs. Graph-Based Framing for Bilingual Lexicon Induction from Word Embedding Spaces](https://mlsanity.com/paper/2109.12640/) (2021)
- [XPASC: Measuring Generalization in Weak Supervision by Explainability and Association](https://mlsanity.com/paper/2206.01444/) (2022)
- [Cross-lingual Transfer for Text Classification with Dictionary-based Heterogeneous Graph](https://mlsanity.com/paper/2109.04400/) (2021)

## Abstract (arXiv)

Cross-lingual word embeddings (CLWE) are often evaluated on bilingual lexicon induction (BLI). Recent CLWE methods use linear projections, which underfit the training dictionary, to generalize on BLI. However, underfitting can hinder generalization to other downstream tasks that rely on words from the training dictionary. We address this limitation by retrofitting CLWE to the training dictionary, which pulls training translation pairs closer in the embedding space and overfits the training dictionary. This simple post-processing step often improves accuracy on two downstream tasks, despite lowering BLI test accuracy. We also retrofit to both the training dictionary and a synthetic dictionary induced from CLWE, which sometimes generalizes even better on downstream tasks. Our results confirm the importance of fully exploiting training dictionary in downstream tasks and explains why BLI is a flawed CLWE evaluation.