mlsanityData to 14 Oct 2026

Evaluating Unsupervised Dimensionality Reduction Methods for Pretrained Sentence Embeddings

arXiv 2403.14001 ↗First posted 20 Mar 2024cs.CLcited by 6

Not on the map. Why?

Abstract (arXiv)

Sentence embeddings produced by Pretrained Language Models (PLMs) have received wide attention from the NLP community due to their superior performance when representing texts in numerous downstream applications. However, the high dimensionality of the sentence embeddings produced by PLMs is problematic when representing large numbers of sentences in memory- or compute-constrained devices. As a solution, we evaluate unsupervised dimensionality reduction methods to reduce the dimensionality of sentence embeddings produced by PLMs. Our experimental results show that simple methods such as Principal Component Analysis (PCA) can reduce the dimensionality of sentence embeddings by almost 50%, without incurring a significant loss in performance in multiple downstream tasks. Surprisingly, reducing the dimensionality further improves performance over the original high-dimensional versions for the sentence embeddings produced by some PLMs in some tasks.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Builds on 30 of its 68 references that are in the map. First posted on arXiv 20 Mar 2024.

Builds on

30 of 68 references.

References · 30

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  3. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  4. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,375
  5. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  6. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
  7. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,329
  8. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,173
  9. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 2,031
  10. 2016MS MARCO: A Human Generated MAchine Reading COmprehension Datasetcited by 1,605
  11. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,167
  12. 2016Pruning Filters for Efficient ConvNetscited by 1,143
18 more
  1. 2017SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluationcited by 1,001
  2. 2020MPNet: Masked and Permuted Pre-training for Language Understandingcited by 981
  3. 2018SNIP: Single-shot Network Pruning based on Connection Sensitivitycited by 812
  4. 2019On Measuring Social Biases in Sentence Encoderscited by 398
  5. 2018SentEval: An Evaluation Toolkit for Universal Sentence Representationscited by 294
  6. 2019Measuring Bias in Contextualized Word Representationscited by 276
  7. 2020Measuring and Reducing Gendered Correlations in Pre-trained Modelscited by 224
  8. 2018Large scale distributed neural network training through online distillationcited by 146
  9. 2019On Measuring and Mitigating Biased Inferences of Word Embeddingscited by 112
  10. 2022Gender Bias in Masked Language Models for Multiple Languagescited by 60
  11. 2015Deep Sentence Embedding Using Long Short-Term Memory Networks: Analysis and Application to Information Retrievalcited by 54
  12. 2019Semantic Product Searchcited by 51
  13. 2017Simple and Effective Dimensionality Reduction for Word Embeddingscited by 39
  14. 2017Compressing Word Embeddings via Deep Compositional Code Learningcited by 35
  15. 2021Evaluation of BERT and ALBERT Sentence Embedding Performance on Downstream NLP Taskscited by 32
  16. 2014An Autoencoder Approach to Learning Bilingual Word Representationscited by 7
  17. 2018Exploiting Sentence Embedding for Medical Question Answeringcited by 5
  18. 2016Compressing Neural Language Models by Sparse Word Representationscited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 3; 2025: 2; 2026: 1.

6 papers.

  1. 2025Randomly Removing 50% of Dimensions in Text Embeddings has Minimal Impact on Retrieval and Classification Taskscited by 8
  2. 2024Mapping News Narratives Using LLMs and Narrative-Structured Text Embeddingscited by 2
  3. 2024Dimensionality Reduction in Sentence Transformer Vector Databases with Fast Fourier Transformcited by 1
  4. 2025Factor Augmented Supervised Learning with Text Embeddingscited by 1
  5. 2026Spectral Tempering for Embedding Compression in Dense Passage Retrievalcited by 1
  6. 2024Word Embedding Dimension Reduction via Weakly-Supervised Feature Selectioncited by 0