Extending Multi-Sense Word Embedding to Phrases and Sentences for Unsupervised Semantic Applications
Not on the map. Why?
Abstract (arXiv)
Most unsupervised NLP models represent each word with a single point or single region in semantic space, while the existing multi-sense word embeddings cannot represent longer word sequences like phrases or sentences. We propose a novel embedding method for a text sequence (a phrase or a sentence) where each sequence is represented by a distinct set of multi-mode codebook embeddings to capture different semantic facets of its meaning. The codebook embeddings can be viewed as the cluster centers which summarize the distribution of possibly co-occurring words in a pre-trained word embedding space. We introduce an end-to-end trainable neural model that directly predicts the set of cluster centers from the input text sequence during test time. Our experiments show that the per-sentence codebook embeddings significantly improve the performances in unsupervised sentence similarity and extractive summarization benchmarks. In phrase similarity experiments, we discover that the multi-facet embeddings provide an interpretable semantic representation but do not outperform the single-facet baseline.
Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 44 of its 87 references that are in the map. First posted on arXiv 29 Mar 2021.
Builds on
44 of 87 references.
References · 44
- 2017Attention Is All You Needcited by 59,932
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
- 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
- 2020End-to-End Object Detection with Transformerscited by 7,384
- 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,462
- 2014Sequence to Sequence Learning with Neural Networkscited by 2,974
- 2018Deep contextualized word representationscited by 2,285
- 2016Enriching Word Vectors with Subword Informationcited by 1,578
- 2017Get To The Point: Summarization with Pointer-Generator Networkscited by 1,281
- 2015Teaching Machines to Read and Comprehendcited by 1,039
- 2017SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluationcited by 998
- 2015Skip-Thought Vectorscited by 321
32 more
- 2020Sparse, Dense, and Attentional Representations for Text Retrievalcited by 213
- 2017Autoencoding Variational Inference For Topic Modelscited by 168
- 2016Linear Algebraic Structure of Word Senses, with Applications to Polysemycited by 165
- 2017Breaking the Softmax Bottleneck: A High-Rank RNN Language Modelcited by 147
- 2015End-to-end people detection in crowded scenescited by 140
- 2018Set Transformer: A Framework for Attention-based Permutation-Invariant Neural Networkscited by 129
- 2019Insertion Transformer: Flexible Sequence Generation via Insertion Operationscited by 123
- 2017Unsupervised Learning of Sentence Embeddings using Compositional n-Gram Featurescited by 103
- 2014Word Representations via Gaussian Embeddingcited by 96
- 2019Sentence Centrality Revisited for Unsupervised Summarizationcited by 64
- 2019Non-Monotonic Sequential Text Generationcited by 55
- 2018Deep Communicating Agents for Abstractive Summarizationcited by 50
- 2015Sparse Overcomplete Word Vector Representationscited by 43
- 2019Insertion-based Decoding with automatically Inferred Generation Ordercited by 43
- 2015Efficient Non-parametric Estimation of Multiple Embeddings per Word in Vector Spacecited by 35
- 2017Compressing Word Embeddings via Deep Compositional Code Learningcited by 34
- 2019L2G Auto-encoder: Understanding Point Clouds by Local-to-Global Reconstruction with Hierarchical Self-Attentioncited by 32
- 2016Improving Hypernymy Detection with an Integrated Path-based and Distributional Methodcited by 26
- 2018Von Mises-Fisher Loss for Training Sequence to Sequence Models with Continuous Outputscited by 26
- 2016Ask the GRU: Multi-Task Learning for Deep Text Recommendationscited by 24
- 2013Domain and Function: A Dual-Space Model of Semantic Relations and Compositionscited by 13
- 2017Multimodal Word Distributionscited by 12
- 2015Multilingual Relation Extraction using Compositional Universal Schemacited by 11
- 2018Context Mover's Distance & Barycenters: Optimal Transport of Contexts for Building Representationscited by 11
- 2014Evaluating Neural Word Representations in Tensor-Based Compositional Settingscited by 7
- 2018Jointly Embedding Entities and Text with Distant Supervisioncited by 7
- 2019Adapting RNN Sequence Prediction Model to Multi-label Set Predictioncited by 7
- 2017Loss Functions for Multiset Predictioncited by 6
- 2017Learning Phrase Embeddings from Paraphrases with GRUscited by 4
- 2020P-SIF: Document Embeddings Using Partition Averagingcited by 4
- 2017Attention-based Mixture Density Recurrent Networks for History-based Recommendationcited by 3
- 2019Improving Document Classification with Multi-Sense Embeddingscited by 3
Built on by
Citing papers per year: 2021: 2; 2022: 1.
3 papers.