# Single Training Dimension Selection for Word Embedding with PCA

arXiv: https://arxiv.org/abs/1909.01761 · HTML: https://mlsanity.com/paper/1909.01761/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 30 Aug 2019.

## Built on by (most cited first, 9 of 9)

- [Graph Entropy Guided Node Embedding Dimension Selection for Graph Neural Networks](https://mlsanity.com/paper/2105.03178/) (2021)
- [Explainability in Practice: A Survey of Explainable NLP Across Various Domains](https://mlsanity.com/paper/2502.00837/) (2025)
- [Preliminary Steps Towards Federated Sentiment Classification](https://mlsanity.com/paper/2107.11956/) (2021)
- [Optimization of embeddings storage for RAG systems using quantization and dimensionality reduction techniques](https://mlsanity.com/paper/2505.00105/) (2025)
- [Compressing Large Language Models with PCA Without Performance Loss](https://mlsanity.com/paper/2508.04307/) (2025)
- [Factor Augmented Supervised Learning with Text Embeddings](https://mlsanity.com/paper/2508.06548/) (2025)
- [Frequency-aware Dimension Selection for Static Word Embedding by Mixed Product Distance](https://mlsanity.com/paper/2305.07826/) (2023)
- [The Impact of Dimensionality on the Stability of Node Embeddings](https://mlsanity.com/paper/2604.08492/) (2026)
- [The Latent Diagnostic Taxonomy: A Framework for Constructing Classifiers and Diagnosing Their Decisions, Applied to Prompt Injection Detection](https://mlsanity.com/paper/2608.26423/) (2026)

## Abstract (arXiv)

In this paper, we present a fast and reliable method based on PCA to select the number of dimensions for word embeddings. First, we train one embedding with a generous upper bound (e.g. 1,000) of dimensions. Then we transform the embeddings using PCA and incrementally remove the lesser dimensions one at a time while recording the embeddings' performance on language tasks. Lastly, we select the number of dimensions while balancing model size and accuracy. Experiments using various datasets and language tasks demonstrate that we are able to train 10 times fewer sets of embeddings while retaining optimal performance. Researchers interested in training the best-performing embeddings for downstream tasks, such as sentiment analysis, question answering and hypernym extraction, as well as those interested in embedding compression should find the method helpful.