# Clustering and Network Analysis for the Embedding Spaces of Sentences and Sub-Sentences

arXiv: https://arxiv.org/abs/2110.00697 · HTML: https://mlsanity.com/paper/2110.00697/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 16 of its 35 references that are in the map.
- First posted on arXiv 2 Oct 2021.

## Builds on (most cited first, 16 of 16)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [Bag of Tricks for Efficient Text Classification](https://mlsanity.com/paper/1607.01759/) (2016)
- [Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyond](https://mlsanity.com/paper/1812.10464/) (2018)
- [Supervised Learning of Universal Sentence Representations from Natural Language Inference Data](https://mlsanity.com/paper/1705.02364/) (2017)
- [Skip-Thought Vectors](https://mlsanity.com/paper/1506.06726/) (2015)
- [Fine-grained Analysis of Sentence Embeddings Using Auxiliary Prediction Tasks](https://mlsanity.com/paper/1608.04207/) (2016)
- [An efficient framework for learning sentence representations](https://mlsanity.com/paper/1803.02893/) (2018)
- [Sentence Embedding Alignment for Lifelong Relation Extraction](https://mlsanity.com/paper/1903.02588/) (2019)
- [CoType: Joint Extraction of Typed Entities and Relations with Knowledge Bases](https://mlsanity.com/paper/1610.08763/) (2016)
- [Evaluation of sentence embeddings in downstream and linguistic probing tasks](https://mlsanity.com/paper/1806.06259/) (2018)
- [The RepEval 2017 Shared Task: Multi-Genre Natural Language Inference with Sentence Representations](https://mlsanity.com/paper/1707.08172/) (2017)
- [To Cluster, or Not to Cluster: An Analysis of Clusterability Methods](https://mlsanity.com/paper/1808.08317/) (2018)
- [Parameter-free Sentence Embedding via Orthogonal Basis](https://mlsanity.com/paper/1810.00438/) (2018)
- [Efficient Sentence Embedding using Discrete Cosine Transform](https://mlsanity.com/paper/1909.03104/) (2019)

## Abstract (arXiv)

Sentence embedding methods offer a powerful approach for working with short textual constructs or sequences of words. By representing sentences as dense numerical vectors, many natural language processing (NLP) applications have improved their performance. However, relatively little is understood about the latent structure of sentence embeddings. Specifically, research has not addressed whether the length and structure of sentences impact the sentence embedding space and topology. This paper reports research on a set of comprehensive clustering and network analyses targeting sentence and sub-sentence embedding spaces. Results show that one method generates the most clusterable embeddings. In general, the embeddings of span sub-sentences have better clustering properties than the original sentences. The results have implications for future sentence embedding models and applications.