# Simple and Effective Dimensionality Reduction for Word Embeddings

arXiv: https://arxiv.org/abs/1708.03629 · HTML: https://mlsanity.com/paper/1708.03629/

## Key facts

- Cited by 39 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 Aug 2017.

## Built on by (most cited first, 30 of 39)

- [Analyzing And Editing Inner Mechanisms Of Backdoored Language Models](https://mlsanity.com/paper/2302.12461/) (2023)
- [Empirical Evaluation of Pre-trained Transformers for Human-Level NLP: The Role of Sample Size and Dimensionality](https://mlsanity.com/paper/2105.03484/) (2021)
- [Prompt-Agnostic Adversarial Perturbation for Customized Diffusion Models](https://mlsanity.com/paper/2408.10571/) (2024)
- [Low Anisotropy Sense Retrofitting (LASeR) : Towards Isotropic and Sense Enriched Representations](https://mlsanity.com/paper/2104.10833/) (2021)
- [Dimension Reduction with Locally Adjusted Graphs](https://mlsanity.com/paper/2412.15426/) (2024)
- [Randomly Removing 50% of Dimensions in Text Embeddings has Minimal Impact on Retrieval and Classification Tasks](https://mlsanity.com/paper/2508.17744/) (2025)
- [When Word Embeddings Become Endangered](https://mlsanity.com/paper/2103.13275/) (2021)
- [Exploring Dimensionality Reduction Techniques in Multilingual Transformers](https://mlsanity.com/paper/2204.08415/) (2022)
- [Evaluating Unsupervised Dimensionality Reduction Methods for Pretrained Sentence Embeddings](https://mlsanity.com/paper/2403.14001/) (2024)
- [Compressing Sentence Representation for Semantic Retrieval via Homomorphic Projective Distillation](https://mlsanity.com/paper/2203.07687/) (2022)
- [Conflict-Aware Pseudo Labeling via Optimal Transport for Entity Alignment](https://mlsanity.com/paper/2209.01847/) (2022)
- [Effects of Pre- and Post-Processing on type-based Embeddings in Lexical Semantic Change Detection](https://mlsanity.com/paper/2101.09368/) (2021)
- [Leveraging cross-platform data to improve automated hate speech detection](https://mlsanity.com/paper/2102.04895/) (2021)
- [The Future is Sparse: Embedding Compression for Scalable Retrieval in Recommender Systems](https://mlsanity.com/paper/2505.11388/) (2025)
- [Navigating the Effect of Parametrization for Dimensionality Reduction](https://mlsanity.com/paper/2411.15894/) (2024)
- [Discrete Speech Unit Extraction via Independent Component Analysis](https://mlsanity.com/paper/2501.06562/) (2025)
- [On Geodesic Distances and Contextual Embedding Compression for Text Classification](https://mlsanity.com/paper/2104.11295/) (2021)
- [Assessment of contextualised representations in detecting outcome phrases in clinical trials](https://mlsanity.com/paper/2203.03547/) (2022)
- [Evaluation of taxonomic and neural embedding methods for calculating semantic similarity](https://mlsanity.com/paper/2209.15197/) (2022)
- [Combating Confirmation Bias: A Unified Pseudo-Labeling Framework for Entity Alignment](https://mlsanity.com/paper/2307.02075/) (2023)
- [Sig-Networks Toolkit: Signature Networks for Longitudinal Language Modelling](https://mlsanity.com/paper/2312.03523/) (2023)
- [A Large-Language-Model Framework for Automated Humanitarian Situation Reporting](https://mlsanity.com/paper/2512.19475/) (2025)
- [On the Effectiveness of Proposed Techniques to Reduce Energy Consumption in RAG Systems: A Controlled Experiment](https://mlsanity.com/paper/2601.02522/) (2026)
- [Preliminary Steps Towards Federated Sentiment Classification](https://mlsanity.com/paper/2107.11956/) (2021)
- [Word Tour: One-dimensional Word Embeddings via the Traveling Salesman Problem](https://mlsanity.com/paper/2205.01954/) (2022)
- [Dimensionality Reduction in Sentence Transformer Vector Databases with Fast Fourier Transform](https://mlsanity.com/paper/2404.06278/) (2024)
- [Breaking Thought Patterns: A Multi-Dimensional Reasoning Framework for LLMs](https://mlsanity.com/paper/2506.13192/) (2025)
- [InsurTech innovation using natural language processing](https://mlsanity.com/paper/2507.21112/) (2025)
- [Pooling and Semantic Shift: The Fundamental Challenges in Long Text Embedding and Retrieval](https://mlsanity.com/paper/2603.21437/) (2026)
- [Data-Driven Evaluation of Training Action Space for Reinforcement Learning](https://mlsanity.com/paper/2204.03840/) (2022)

## Abstract (arXiv)

Word embeddings have become the basic building blocks for several natural language processing and information retrieval tasks. Pre-trained word embeddings are used in several downstream applications as well as for constructing representations for sentences, paragraphs and documents. Recently, there has been an emphasis on further improving the pre-trained word vectors through post-processing algorithms. One such area of improvement is the dimensionality reduction of the word embeddings. Reducing the size of word embeddings through dimensionality reduction can improve their utility in memory constrained devices, benefiting several real-world applications. In this work, we present a novel algorithm that effectively combines PCA based dimensionality reduction with a recently proposed post-processing algorithm, to construct word embeddings of lower dimensions. Empirical evaluations on 12 standard word similarity benchmarks show that our algorithm reduces the embedding dimensionality by 50%, while achieving similar or (more often) better performance than the higher dimension embeddings.