# Evaluating Unsupervised Dimensionality Reduction Methods for Pretrained Sentence Embeddings

arXiv: https://arxiv.org/abs/2403.14001 · HTML: https://mlsanity.com/paper/2403.14001/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 68 references that are in the map.
- First posted on arXiv 20 Mar 2024.

## Builds on (most cited first, 30 of 30)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding](https://mlsanity.com/paper/1510.00149/) (2015)
- [SimCSE: Simple Contrastive Learning of Sentence Embeddings](https://mlsanity.com/paper/2104.08821/) (2021)
- [Learning both Weights and Connections for Efficient Neural Networks](https://mlsanity.com/paper/1506.02626/) (2015)
- [MS MARCO: A Human Generated MAchine Reading COmprehension Dataset](https://mlsanity.com/paper/1611.09268/) (2016)
- [TinyBERT: Distilling BERT for Natural Language Understanding](https://mlsanity.com/paper/1909.10351/) (2019)
- [Pruning Filters for Efficient ConvNets](https://mlsanity.com/paper/1608.08710/) (2016)
- [SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluation](https://mlsanity.com/paper/1708.00055/) (2017)
- [MPNet: Masked and Permuted Pre-training for Language Understanding](https://mlsanity.com/paper/2004.09297/) (2020)
- [SNIP: Single-shot Network Pruning based on Connection Sensitivity](https://mlsanity.com/paper/1810.02340/) (2018)
- [On Measuring Social Biases in Sentence Encoders](https://mlsanity.com/paper/1903.10561/) (2019)
- [SentEval: An Evaluation Toolkit for Universal Sentence Representations](https://mlsanity.com/paper/1803.05449/) (2018)
- [Measuring Bias in Contextualized Word Representations](https://mlsanity.com/paper/1906.07337/) (2019)
- [Measuring and Reducing Gendered Correlations in Pre-trained Models](https://mlsanity.com/paper/2010.06032/) (2020)
- [Large scale distributed neural network training through online distillation](https://mlsanity.com/paper/1804.03235/) (2018)
- [On Measuring and Mitigating Biased Inferences of Word Embeddings](https://mlsanity.com/paper/1908.09369/) (2019)
- [Gender Bias in Masked Language Models for Multiple Languages](https://mlsanity.com/paper/2205.00551/) (2022)
- [Deep Sentence Embedding Using Long Short-Term Memory Networks: Analysis and Application to Information Retrieval](https://mlsanity.com/paper/1502.06922/) (2015)
- [Semantic Product Search](https://mlsanity.com/paper/1907.00937/) (2019)
- [Simple and Effective Dimensionality Reduction for Word Embeddings](https://mlsanity.com/paper/1708.03629/) (2017)
- [Compressing Word Embeddings via Deep Compositional Code Learning](https://mlsanity.com/paper/1711.01068/) (2017)
- [Evaluation of BERT and ALBERT Sentence Embedding Performance on Downstream NLP Tasks](https://mlsanity.com/paper/2101.10642/) (2021)
- [An Autoencoder Approach to Learning Bilingual Word Representations](https://mlsanity.com/paper/1402.1454/) (2014)
- [Exploiting Sentence Embedding for Medical Question Answering](https://mlsanity.com/paper/1811.06156/) (2018)
- [Compressing Neural Language Models by Sparse Word Representations](https://mlsanity.com/paper/1610.03950/) (2016)

## Built on by (most cited first, 6 of 6)

- [Randomly Removing 50% of Dimensions in Text Embeddings has Minimal Impact on Retrieval and Classification Tasks](https://mlsanity.com/paper/2508.17744/) (2025)
- [Mapping News Narratives Using LLMs and Narrative-Structured Text Embeddings](https://mlsanity.com/paper/2409.06540/) (2024)
- [Dimensionality Reduction in Sentence Transformer Vector Databases with Fast Fourier Transform](https://mlsanity.com/paper/2404.06278/) (2024)
- [Factor Augmented Supervised Learning with Text Embeddings](https://mlsanity.com/paper/2508.06548/) (2025)
- [Spectral Tempering for Embedding Compression in Dense Passage Retrieval](https://mlsanity.com/paper/2603.19339/) (2026)
- [Word Embedding Dimension Reduction via Weakly-Supervised Feature Selection](https://mlsanity.com/paper/2407.12342/) (2024)

## Abstract (arXiv)

Sentence embeddings produced by Pretrained Language Models (PLMs) have received wide attention from the NLP community due to their superior performance when representing texts in numerous downstream applications. However, the high dimensionality of the sentence embeddings produced by PLMs is problematic when representing large numbers of sentences in memory- or compute-constrained devices. As a solution, we evaluate unsupervised dimensionality reduction methods to reduce the dimensionality of sentence embeddings produced by PLMs. Our experimental results show that simple methods such as Principal Component Analysis (PCA) can reduce the dimensionality of sentence embeddings by almost 50%, without incurring a significant loss in performance in multiple downstream tasks. Surprisingly, reducing the dimensionality further improves performance over the original high-dimensional versions for the sentence embeddings produced by some PLMs in some tasks.