# Extending Multi-Sense Word Embedding to Phrases and Sentences for Unsupervised Semantic Applications

arXiv: https://arxiv.org/abs/2103.15330 · HTML: https://mlsanity.com/paper/2103.15330/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 44 of its 87 references that are in the map.
- First posted on arXiv 29 Mar 2021.

## Builds on (most cited first, 30 of 44)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [Sequence to Sequence Learning with Neural Networks](https://mlsanity.com/paper/1409.3215/) (2014)
- [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018)
- [Enriching Word Vectors with Subword Information](https://mlsanity.com/paper/1607.04606/) (2016)
- [Get To The Point: Summarization with Pointer-Generator Networks](https://mlsanity.com/paper/1704.04368/) (2017)
- [Teaching Machines to Read and Comprehend](https://mlsanity.com/paper/1506.03340/) (2015)
- [SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluation](https://mlsanity.com/paper/1708.00055/) (2017)
- [Skip-Thought Vectors](https://mlsanity.com/paper/1506.06726/) (2015)
- [Sparse, Dense, and Attentional Representations for Text Retrieval](https://mlsanity.com/paper/2005.00181/) (2020)
- [Autoencoding Variational Inference For Topic Models](https://mlsanity.com/paper/1703.01488/) (2017)
- [Linear Algebraic Structure of Word Senses, with Applications to Polysemy](https://mlsanity.com/paper/1601.03764/) (2016)
- [Breaking the Softmax Bottleneck: A High-Rank RNN Language Model](https://mlsanity.com/paper/1711.03953/) (2017)
- [End-to-end people detection in crowded scenes](https://mlsanity.com/paper/1506.04878/) (2015)
- [Set Transformer: A Framework for Attention-based Permutation-Invariant Neural Networks](https://mlsanity.com/paper/1810.00825/) (2018)
- [Insertion Transformer: Flexible Sequence Generation via Insertion Operations](https://mlsanity.com/paper/1902.03249/) (2019)
- [Unsupervised Learning of Sentence Embeddings using Compositional n-Gram Features](https://mlsanity.com/paper/1703.02507/) (2017)
- [Word Representations via Gaussian Embedding](https://mlsanity.com/paper/1412.6623/) (2014)
- [Sentence Centrality Revisited for Unsupervised Summarization](https://mlsanity.com/paper/1906.03508/) (2019)
- [Non-Monotonic Sequential Text Generation](https://mlsanity.com/paper/1902.02192/) (2019)
- [Deep Communicating Agents for Abstractive Summarization](https://mlsanity.com/paper/1803.10357/) (2018)
- [Sparse Overcomplete Word Vector Representations](https://mlsanity.com/paper/1506.02004/) (2015)
- [Insertion-based Decoding with automatically Inferred Generation Order](https://mlsanity.com/paper/1902.01370/) (2019)
- [Efficient Non-parametric Estimation of Multiple Embeddings per Word in Vector Space](https://mlsanity.com/paper/1504.06654/) (2015)
- [Compressing Word Embeddings via Deep Compositional Code Learning](https://mlsanity.com/paper/1711.01068/) (2017)
- [L2G Auto-encoder: Understanding Point Clouds by Local-to-Global Reconstruction with Hierarchical Self-Attention](https://mlsanity.com/paper/1908.00720/) (2019)
- [Improving Hypernymy Detection with an Integrated Path-based and Distributional Method](https://mlsanity.com/paper/1603.06076/) (2016)

## Built on by (most cited first, 3 of 3)

- [SciCo: Hierarchical Cross-Document Coreference for Scientific Concepts](https://mlsanity.com/paper/2104.08809/) (2021)
- [Multi-CLS BERT: An Efficient Alternative to Traditional Ensembling](https://mlsanity.com/paper/2210.05043/) (2022)
- [Trust It or Not: Confidence-Guided Automatic Radiology Report Generation](https://mlsanity.com/paper/2106.10887/) (2021)

## Abstract (arXiv)

Most unsupervised NLP models represent each word with a single point or single region in semantic space, while the existing multi-sense word embeddings cannot represent longer word sequences like phrases or sentences. We propose a novel embedding method for a text sequence (a phrase or a sentence) where each sequence is represented by a distinct set of multi-mode codebook embeddings to capture different semantic facets of its meaning. The codebook embeddings can be viewed as the cluster centers which summarize the distribution of possibly co-occurring words in a pre-trained word embedding space. We introduce an end-to-end trainable neural model that directly predicts the set of cluster centers from the input text sequence during test time. Our experiments show that the per-sentence codebook embeddings significantly improve the performances in unsupervised sentence similarity and extractive summarization benchmarks. In phrase similarity experiments, we discover that the multi-facet embeddings provide an interpretable semantic representation but do not outperform the single-facet baseline.