# Denoising Word Embeddings by Averaging in a Shared Space

arXiv: https://arxiv.org/abs/2106.02954 · HTML: https://mlsanity.com/paper/2106.02954/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 46 references that are in the map.
- First posted on arXiv 5 Jun 2021.

## Builds on (most cited first, 19 of 19)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018)
- [Enriching Word Vectors with Subword Information](https://mlsanity.com/paper/1607.04606/) (2016)
- [Word Translation Without Parallel Data](https://mlsanity.com/paper/1710.04087/) (2017)
- [Diachronic Word Embeddings Reveal Statistical Laws of Semantic Change](https://mlsanity.com/paper/1605.09096/) (2016)
- [SimLex-999: Evaluating Semantic Models with (Genuine) Similarity Estimation](https://mlsanity.com/paper/1408.3456/) (2014)
- [Unsupervised Neural Machine Translation](https://mlsanity.com/paper/1710.11041/) (2017)
- [A robust self-learning method for fully unsupervised cross-lingual mappings of word embeddings](https://mlsanity.com/paper/1805.06297/) (2018)
- [Learning Multilingual Word Embeddings in Latent Metric Space: A Geometric Approach](https://mlsanity.com/paper/1808.08773/) (2018)
- [Unsupervised Hyperalignment for Multilingual Word Embeddings](https://mlsanity.com/paper/1811.01124/) (2018)
- [CDLM: Cross-Document Language Modeling](https://mlsanity.com/paper/2101.00406/) (2021)
- [Unsupervised Multilingual Word Embeddings](https://mlsanity.com/paper/1808.08933/) (2018)
- [Frustratingly Easy Meta-Embedding -- Computing Meta-Embeddings by Averaging Source Word Embeddings](https://mlsanity.com/paper/1804.05262/) (2018)
- [Dynamic Meta-Embeddings for Improved Sentence Representations](https://mlsanity.com/paper/1804.07983/) (2018)
- [Paraphrasing vs Coreferring: Two Sides of the Same Coin](https://mlsanity.com/paper/2004.14979/) (2020)
- [Bayesian Hierarchical Words Representation Learning](https://mlsanity.com/paper/2004.07126/) (2020)
- [A Discriminative Latent-Variable Model for Bilingual Lexicon Induction](https://mlsanity.com/paper/1808.09334/) (2018)
- [Linear Ensembles of Word Embedding Models](https://mlsanity.com/paper/1704.01419/) (2017)

## Built on by (most cited first, 2 of 2)

- [Representation Learning via Variational Bayesian Networks](https://mlsanity.com/paper/2306.16326/) (2023)
- [TransDrift: Modeling Word-Embedding Drift using Transformer](https://mlsanity.com/paper/2206.08081/) (2022)

## Abstract (arXiv)

We introduce a new approach for smoothing and improving the quality of word embeddings. We consider a method of fusing word embeddings that were trained on the same corpus but with different initializations. We project all the models to a shared vector space using an efficient implementation of the Generalized Procrustes Analysis (GPA) procedure, previously used in multilingual word translation. Our word representation demonstrates consistent improvements over the raw models as well as their simplistic average, on a range of tasks. As the new representations are more stable and reliable, there is a noticeable improvement in rare word evaluations.