# Hubness Reduction Improves Sentence-BERT Semantic Spaces

arXiv: https://arxiv.org/abs/2311.18364 · HTML: https://mlsanity.com/paper/2311.18364/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 32 references that are in the map.
- First posted on arXiv 30 Nov 2023.

## Builds on (most cited first, 13 of 13)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [Dense Passage Retrieval for Open-Domain Question Answering](https://mlsanity.com/paper/2004.04906/) (2020)
- [Character-level Convolutional Networks for Text Classification](https://mlsanity.com/paper/1509.01626/) (2015)
- [MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers](https://mlsanity.com/paper/2002.10957/) (2020)
- [MPNet: Masked and Permuted Pre-training for Language Understanding](https://mlsanity.com/paper/2004.09297/) (2020)
- [Efficient and robust approximate nearest neighbor search using Hierarchical Navigable Small World graphs](https://mlsanity.com/paper/1603.09320/) (2016)
- [All Bark and No Bite: Rogue Dimensions in Transformer Language Models Obscure Representational Quality](https://mlsanity.com/paper/2109.04404/) (2021)
- [Cross Modal Retrieval with Querybank Normalisation](https://mlsanity.com/paper/2112.12777/) (2021)
- [Cross-lingual Information Retrieval with BERT](https://mlsanity.com/paper/2004.13005/) (2020)
- [Analyzing the Limitations of Cross-lingual Word Embedding Mappings](https://mlsanity.com/paper/1906.05407/) (2019)
- [scikit-hubness: Hubness Reduction and Approximate Neighbor Search](https://mlsanity.com/paper/1912.00706/) (2019)

## Built on by (most cited first, 4 of 4)

- [Prediction hubs are context-informed frequent tokens in LLMs](https://mlsanity.com/paper/2502.10201/) (2025)
- [OntoLearner: A Modular Python Library for Ontology Learning with Large Language Models](https://mlsanity.com/paper/2607.01977/) (2026)
- [The Crowded Embedding Space: A Mean-Field Mechanism for Emergent Marginalization in Retrieval-Augmented Agents](https://mlsanity.com/paper/2606.28343/) (2026)
- [A Manifold-Aware Topic Modeling Approach via Rank-Based Prototypes](https://mlsanity.com/paper/2609.29630/) (2026)

## Abstract (arXiv)

Semantic representations of text, i.e. representations of natural language which capture meaning by geometry, are essential for areas such as information retrieval and document grouping. High-dimensional trained dense vectors have received much attention in recent years as such representations. We investigate the structure of semantic spaces that arise from embeddings made with Sentence-BERT and find that the representations suffer from a well-known problem in high dimensions called hubness. Hubness results in asymmetric neighborhood relations, such that some texts (the hubs) are neighbours of many other texts while most texts (so-called anti-hubs), are neighbours of few or no other texts. We quantify the semantic quality of the embeddings using hubness scores and error rate of a neighbourhood based classifier. We find that when hubness is high, we can reduce error rate and hubness using hubness reduction methods. We identify a combination of two methods as resulting in the best reduction. For example, on one of the tested pretrained models, this combined method can reduce hubness by about 75% and error rate by about 9%. Thus, we argue that mitigating hubness in the embedding space provides better semantic representations of text.