# On the Dimensionality of Word Embedding

arXiv: https://arxiv.org/abs/1812.04224 · HTML: https://mlsanity.com/paper/1812.04224/

## Key facts

- Cited by 46 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 Dec 2018.

## Built on by (most cited first, 30 of 46)

- [Similarity of Neural Network Models: A Survey of Functional and Representational Measures](https://mlsanity.com/paper/2305.06329/) (2023)
- [On the Theoretical Limitations of Embedding-Based Retrieval](https://mlsanity.com/paper/2508.21038/) (2025)
- [Multi-Task Model Merging via Adaptive Weight Disentanglement](https://mlsanity.com/paper/2411.18729/) (2024)
- [ReSi: A Comprehensive Benchmark for Representational Similarity Measures](https://mlsanity.com/paper/2408.00531/) (2024)
- [Graph Entropy Guided Node Embedding Dimension Selection for Graph Neural Networks](https://mlsanity.com/paper/2105.03178/) (2021)
- [NeuroCartography: Scalable Automatic Visual Summarization of Concepts in Deep Neural Networks](https://mlsanity.com/paper/2108.12931/) (2021)
- [Temporal Aggregation and Propagation Graph Neural Networks for Dynamic Representation](https://mlsanity.com/paper/2304.07503/) (2023)
- [A Survey on Word Meta-Embedding Learning](https://mlsanity.com/paper/2204.11660/) (2022)
- [Frequency-aware SGD for Efficient Embedding Learning with Provable Benefits](https://mlsanity.com/paper/2110.04844/) (2021)
- [Towards Measuring Representational Similarity of Large Language Models](https://mlsanity.com/paper/2312.02730/) (2023)
- [Learning Meta Word Embeddings by Unsupervised Weighted Concatenation of Source Embeddings](https://mlsanity.com/paper/2204.12386/) (2022)
- [Reliable Measures of Spread in High Dimensional Latent Spaces](https://mlsanity.com/paper/2212.08172/) (2022)
- [On the Dimensionality of Sentence Embeddings](https://mlsanity.com/paper/2310.15285/) (2023)
- [What Limits Does Quantization Place on Dense Top-k Retrieval? A Theoretical Study](https://mlsanity.com/paper/2606.11780/) (2026)
- [Tiny language models](https://mlsanity.com/paper/2507.14871/) (2025)
- [Evaluation of taxonomic and neural embedding methods for calculating semantic similarity](https://mlsanity.com/paper/2209.15197/) (2022)
- [Generating Human Understandable Explanations for Node Embeddings](https://mlsanity.com/paper/2406.07642/) (2024)
- [Appformer: A Novel Framework for Mobile App Usage Prediction Leveraging Progressive Multi-Modal Data Fusion and Feature Extraction](https://mlsanity.com/paper/2407.19414/) (2024)
- [Learning Mechanism Underlying NLP Pre-Training and Fine-Tuning](https://mlsanity.com/paper/2509.03407/) (2025)
- [Manifold Approximation leads to Robust Kernel Alignment](https://mlsanity.com/paper/2510.22953/) (2025)
- [Effects of Number of Filters of Convolutional Layers on Speech Recognition Model Accuracy](https://mlsanity.com/paper/2102.02326/) (2021)
- [Micro-level Reserving for General Insurance Claims using a Long Short-Term Memory Network](https://mlsanity.com/paper/2201.13267/) (2022)
- [Statistical process monitoring of artificial neural networks](https://mlsanity.com/paper/2209.07436/) (2022)
- [Together We Make Sense -- Learning Meta-Sense Embeddings from Pretrained Static Sense Embeddings](https://mlsanity.com/paper/2305.19092/) (2023)
- [Relative Overfitting and Accept-Reject Framework](https://mlsanity.com/paper/2505.07783/) (2025)
- [Embeddings to Diagnosis: Latent Fragility under Agentic Perturbations in Clinical LLMs](https://mlsanity.com/paper/2507.21188/) (2025)
- [Factor Augmented Supervised Learning with Text Embeddings](https://mlsanity.com/paper/2508.06548/) (2025)
- [SEMIE: SEMantically Infused Embeddings with Enhanced Interpretability for Domain-specific Small Corpus](https://mlsanity.com/paper/2103.11431/) (2021)
- [CRL+: A Novel Semi-Supervised Deep Active Contrastive Representation Learning-Based Text Classification Model for Insurance Data](https://mlsanity.com/paper/2302.04343/) (2023)
- [Frequency-aware Dimension Selection for Static Word Embedding by Mixed Product Distance](https://mlsanity.com/paper/2305.07826/) (2023)

## Abstract (arXiv)

In this paper, we provide a theoretical understanding of word embedding and its dimensionality. Motivated by the unitary-invariance of word embedding, we propose the Pairwise Inner Product (PIP) loss, a novel metric on the dissimilarity between word embeddings. Using techniques from matrix perturbation theory, we reveal a fundamental bias-variance trade-off in dimensionality selection for word embeddings. This bias-variance trade-off sheds light on many empirical observations which were previously unexplained, for example the existence of an optimal dimensionality. Moreover, new insights and discoveries, like when and how word embeddings are robust to over-fitting, are revealed. By optimizing over the bias-variance trade-off of the PIP loss, we can explicitly answer the open question of dimensionality selection for word embedding.