# Together We Make Sense -- Learning Meta-Sense Embeddings from Pretrained Static Sense Embeddings

arXiv: https://arxiv.org/abs/2305.19092 · HTML: https://mlsanity.com/paper/2305.19092/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 49 references that are in the map.
- First posted on arXiv 30 May 2023.

## Builds on (most cited first, 25 of 25)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [WiC: the Word-in-Context Dataset for Evaluating Context-Sensitive Meaning Representations](https://mlsanity.com/paper/1808.09121/) (2018)
- [SenseBERT: Driving Some Sense into BERT](https://mlsanity.com/paper/1908.05646/) (2019)
- [From Word to Sense Embeddings: A Survey on Vector Representations of Meaning](https://mlsanity.com/paper/1805.04032/) (2018)
- [On the Dimensionality of Word Embedding](https://mlsanity.com/paper/1812.04224/) (2018)
- [XL-WiC: A Multilingual Benchmark for Evaluating Semantic Contextualization](https://mlsanity.com/paper/2010.06478/) (2020)
- [Language Modelling Makes Sense: Propagating Representations through WordNet for Full-Coverage Word Sense Disambiguation](https://mlsanity.com/paper/1906.10007/) (2019)
- [Efficient Non-parametric Estimation of Multiple Embeddings per Word in Vector Space](https://mlsanity.com/paper/1504.06654/) (2015)
- [AutoExtend: Extending Word Embeddings to Embeddings for Synsets and Lexemes](https://mlsanity.com/paper/1507.01127/) (2015)
- [Sense Embeddings are also Biased--Evaluating Social Biases in Static and Contextualised Sense Embeddings](https://mlsanity.com/paper/2203.07523/) (2022)
- [LMMS Reloaded: Transformer-based Sense Embeddings for Disambiguation and Beyond](https://mlsanity.com/paper/2105.12449/) (2021)
- [Frustratingly Easy Meta-Embedding -- Computing Meta-Embeddings by Averaging Source Word Embeddings](https://mlsanity.com/paper/1804.05262/) (2018)
- [Dynamic Meta-Embeddings for Improved Sentence Representations](https://mlsanity.com/paper/1804.07983/) (2018)
- [De-Conflated Semantic Representations](https://mlsanity.com/paper/1608.01961/) (2016)
- [Sentence Meta-Embeddings for Unsupervised Semantic Textual Similarity](https://mlsanity.com/paper/1911.03700/) (2019)
- [LIAAD at SemDeep-5 Challenge: Word-in-Context (WiC)](https://mlsanity.com/paper/1906.10002/) (2019)
- [A Survey on Word Meta-Embedding Learning](https://mlsanity.com/paper/2204.11660/) (2022)
- [Learning Geometric Word Meta-Embeddings](https://mlsanity.com/paper/2004.09219/) (2020)
- [Learning Sense-Specific Static Embeddings using Contextualised Word Embeddings as a Proxy](https://mlsanity.com/paper/2110.02204/) (2021)
- [Gender Bias in Meta-Embeddings](https://mlsanity.com/paper/2205.09867/) (2022)
- [Learning Meta Word Embeddings by Unsupervised Weighted Concatenation of Source Embeddings](https://mlsanity.com/paper/2204.12386/) (2022)
- [Unsupervised Attention-based Sentence-Level Meta-Embeddings from Contextualised Language Models](https://mlsanity.com/paper/2204.07746/) (2022)

## Built on by (most cited first, 1 of 1)

- [OYXOY: A Modern NLP Test Suite for Modern Greek](https://mlsanity.com/paper/2309.07009/) (2023)

## Abstract (arXiv)

Sense embedding learning methods learn multiple vectors for a given ambiguous word, corresponding to its different word senses. For this purpose, different methods have been proposed in prior work on sense embedding learning that use different sense inventories, sense-tagged corpora and learning methods. However, not all existing sense embeddings cover all senses of ambiguous words equally well due to the discrepancies in their training resources. To address this problem, we propose the first-ever meta-sense embedding method -- Neighbour Preserving Meta-Sense Embeddings, which learns meta-sense embeddings by combining multiple independently trained source sense embeddings such that the sense neighbourhoods computed from the source embeddings are preserved in the meta-embedding space. Our proposed method can combine source sense embeddings that cover different sets of word senses. Experimental results on Word Sense Disambiguation (WSD) and Word-in-Context (WiC) tasks show that the proposed meta-sense embedding method consistently outperforms several competitive baselines.