# Learning Meta Word Embeddings by Unsupervised Weighted Concatenation of Source Embeddings

arXiv: https://arxiv.org/abs/2204.12386 · HTML: https://mlsanity.com/paper/2204.12386/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 103 references that are in the map.
- First posted on arXiv 26 Apr 2022.

## Builds on (most cited first, 22 of 22)

- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [Diachronic Word Embeddings Reveal Statistical Laws of Semantic Change](https://mlsanity.com/paper/1605.09096/) (2016)
- [Predictive Biases in Natural Language Processing Models: A Conceptual Framework and Overview](https://mlsanity.com/paper/1912.11078/) (2019)
- [Offline bilingual word vectors, orthogonal transformations and the inverted softmax](https://mlsanity.com/paper/1702.03859/) (2017)
- [Gender-preserving Debiasing for Pre-trained Word Embeddings](https://mlsanity.com/paper/1906.00742/) (2019)
- [On the Dimensionality of Word Embedding](https://mlsanity.com/paper/1812.04224/) (2018)
- [Efficient Non-parametric Estimation of Multiple Embeddings per Word in Vector Space](https://mlsanity.com/paper/1504.06654/) (2015)
- [Problems With Evaluation of Word Embeddings Using Word Similarity Tasks](https://mlsanity.com/paper/1605.02276/) (2016)
- [Frustratingly Easy Meta-Embedding -- Computing Meta-Embeddings by Averaging Source Word Embeddings](https://mlsanity.com/paper/1804.05262/) (2018)
- [Dynamic Meta-Embeddings for Improved Sentence Representations](https://mlsanity.com/paper/1804.07983/) (2018)
- [Domain and Function: A Dual-Space Model of Semantic Relations and Compositions](https://mlsanity.com/paper/1309.4035/) (2013)
- [Adaptive Joint Learning of Compositional and Non-Compositional Phrase Embeddings](https://mlsanity.com/paper/1603.06067/) (2016)
- [Think Globally, Embed Locally --- Locally Linear Meta-embedding of Words](https://mlsanity.com/paper/1709.06671/) (2017)
- [A Survey on Word Meta-Embedding Learning](https://mlsanity.com/paper/2204.11660/) (2022)
- [Learning Geometric Word Meta-Embeddings](https://mlsanity.com/paper/2004.09219/) (2020)
- [SentiWords: Deriving a High Precision and High Coverage Lexicon for Sentiment Analysis](https://mlsanity.com/paper/1510.09079/) (2015)
- [Learning Efficient Task-Specific Meta-Embeddings with Word Prisms](https://mlsanity.com/paper/2011.02944/) (2020)
- [Unsupervised Attention-based Sentence-Level Meta-Embeddings from Contextualised Language Models](https://mlsanity.com/paper/2204.07746/) (2022)
- [A Generative Word Embedding Model and its Low Rank Positive Semidefinite Solution](https://mlsanity.com/paper/1508.03826/) (2015)
- [Learning Meta-Embeddings by Using Ensembles of Embedding Sets](https://mlsanity.com/paper/1508.04257/) (2015)
- [Angular-Based Word Meta-Embedding Learning](https://mlsanity.com/paper/1808.04334/) (2018)

## Built on by (most cited first, 5 of 5)

- [Gender Bias in Meta-Embeddings](https://mlsanity.com/paper/2205.09867/) (2022)
- [A Neighbourhood-Aware Differential Privacy Mechanism for Static Word Embeddings](https://mlsanity.com/paper/2309.10551/) (2023)
- [Together We Make Sense -- Learning Meta-Sense Embeddings from Pretrained Static Sense Embeddings](https://mlsanity.com/paper/2305.19092/) (2023)
- [Map of Encoders -- Mapping Sentence Encoders using Quantum Relative Entropy](https://mlsanity.com/paper/2602.08740/) (2026)
- [CONE: Embeddings for Complex Numerical Data Preserving Unit and Variable Semantics](https://mlsanity.com/paper/2603.04741/) (2026)

## Abstract (arXiv)

Given multiple source word embeddings learnt using diverse algorithms and lexical resources, meta word embedding learning methods attempt to learn more accurate and wide-coverage word embeddings. Prior work on meta-embedding has repeatedly discovered that simple vector concatenation of the source embeddings to be a competitive baseline. However, it remains unclear as to why and when simple vector concatenation can produce accurate meta-embeddings. We show that weighted concatenation can be seen as a spectrum matching operation between each source embedding and the meta-embedding, minimising the pairwise inner-product loss. Following this theoretical analysis, we propose two unsupervised methods to learn the optimal concatenation weights for creating meta-embeddings from a given set of source embeddings. Experimental results on multiple benchmark datasets show that the proposed weighted concatenated meta-embedding methods outperform previously proposed meta-embedding learning methods.