# Unsupervised Sentence-embeddings by Manifold Approximation and Projection

arXiv: https://arxiv.org/abs/2102.03795 · HTML: https://mlsanity.com/paper/2102.03795/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 42 references that are in the map.
- First posted on arXiv 7 Feb 2021.

## Builds on (most cited first, 16 of 16)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction](https://mlsanity.com/paper/1802.03426/) (2018)
- [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018)
- [Distributed Representations of Sentences and Documents](https://mlsanity.com/paper/1405.4053/) (2014)
- [Supervised Learning of Universal Sentence Representations from Natural Language Inference Data](https://mlsanity.com/paper/1705.02364/) (2017)
- [Skip-Thought Vectors](https://mlsanity.com/paper/1506.06726/) (2015)
- [Deep Sentence Embedding Using Long Short-Term Memory Networks: Analysis and Application to Information Retrieval](https://mlsanity.com/paper/1502.06922/) (2015)
- [Word Mover's Embedding: From Word2Vec to Document Embedding](https://mlsanity.com/paper/1811.01713/) (2018)
- [Concatenated Power Mean Word Embeddings as Universal Cross-Lingual Sentence Representations](https://mlsanity.com/paper/1803.01400/) (2018)
- [Efficient Vector Representation for Documents through Corruption](https://mlsanity.com/paper/1707.02377/) (2017)
- [Context Mover's Distance & Barycenters: Optimal Transport of Contexts for Building Representations](https://mlsanity.com/paper/1808.09663/) (2018)
- [Efficient Sentence Embedding using Discrete Cosine Transform](https://mlsanity.com/paper/1909.03104/) (2019)
- [Wasserstein-Fisher-Rao Document Distance](https://mlsanity.com/paper/1904.10294/) (2019)
- [Evaluation of Unsupervised Compositional Representations](https://mlsanity.com/paper/1806.04713/) (2018)

## Built on by (most cited first, 4 of 4)

- [Non-Linguistic Supervision for Contrastive Learning of Sentence Embeddings](https://mlsanity.com/paper/2209.09433/) (2022)
- [Clustering with UMAP: Why and How Connectivity Matters](https://mlsanity.com/paper/2108.05525/) (2021)
- [Extracting Sentence Embeddings from Pretrained Transformer Models](https://mlsanity.com/paper/2408.08073/) (2024)
- [Topological quantification of ambiguity in semantic search](https://mlsanity.com/paper/2406.07990/) (2024)

## Abstract (arXiv)

The concept of unsupervised universal sentence encoders has gained traction recently, wherein pre-trained models generate effective task-agnostic fixed-dimensional representations for phrases, sentences and paragraphs. Such methods are of varying complexity, from simple weighted-averages of word vectors to complex language-models based on bidirectional transformers. In this work we propose a novel technique to generate sentence-embeddings in an unsupervised fashion by projecting the sentences onto a fixed-dimensional manifold with the objective of preserving local neighbourhoods in the original space. To delineate such neighbourhoods we experiment with several set-distance metrics, including the recently proposed Word Mover's distance, while the fixed-dimensional projection is achieved by employing a scalable and efficient manifold approximation method rooted in topological data analysis. We test our approach, which we term EMAP or Embeddings by Manifold Approximation and Projection, on six publicly available text-classification datasets of varying size and complexity. Empirical results show that our method consistently performs similar to or better than several alternative state-of-the-art approaches.