mlsanityData to 7 Oct 2026

On the Dimensionality of Word Embedding

arXiv 1812.04224 ↗First posted 11 Dec 2018cs.LG cs.CL stat.MLcited by 46

Not on the map. Why?

Abstract (arXiv)

In this paper, we provide a theoretical understanding of word embedding and its dimensionality. Motivated by the unitary-invariance of word embedding, we propose the Pairwise Inner Product (PIP) loss, a novel metric on the dissimilarity between word embeddings. Using techniques from matrix perturbation theory, we reveal a fundamental bias-variance trade-off in dimensionality selection for word embeddings. This bias-variance trade-off sheds light on many empirical observations which were previously unexplained, for example the existence of an optimal dimensionality. Moreover, new insights and discoveries, like when and how word embeddings are robust to over-fitting, are revealed. By optimizing over the bias-variance trade-off of the PIP loss, we can explicitly answer the open question of dimensionality selection for word embedding.

Key facts. Cited by 46 papers in the map (data through 7 Oct 2026). First posted on arXiv 11 Dec 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 5; 2022: 6; 2023: 10; 2024: 5; 2025: 11; 2026: 9.

46 papers.

Sort:

By area:

  1. 2023Similarity of Neural Network Models: A Survey of Functional and Representational Measurescited by 128
  2. 2025On the Theoretical Limitations of Embedding-Based Retrievalcited by 98
  3. 2024Multi-Task Model Merging via Adaptive Weight Disentanglementcited by 22
  4. 2024ReSi: A Comprehensive Benchmark for Representational Similarity Measurescited by 18
  5. 2021Graph Entropy Guided Node Embedding Dimension Selection for Graph Neural Networkscited by 13
  6. 2021NeuroCartography: Scalable Automatic Visual Summarization of Concepts in Deep Neural Networkscited by 12
  7. 2023Temporal Aggregation and Propagation Graph Neural Networks for Dynamic Representationcited by 8
  8. 2022A Survey on Word Meta-Embedding Learningcited by 7
  9. 2021Frequency-aware SGD for Efficient Embedding Learning with Provable Benefitscited by 6
  10. 2022Learning Meta Word Embeddings by Unsupervised Weighted Concatenation of Source Embeddingscited by 5
  11. 2022Reliable Measures of Spread in High Dimensional Latent Spacescited by 5
  12. 2023On the Dimensionality of Sentence Embeddingscited by 5
34 more
  1. 2023Towards Measuring Representational Similarity of Large Language Modelscited by 5
  2. 2025Tiny language modelscited by 3
  3. 2026What Limits Does Quantization Place on Dense Top-k Retrieval? A Theoretical Studycited by 3
  4. 2022Evaluation of taxonomic and neural embedding methods for calculating semantic similaritycited by 2
  5. 2024Generating Human Understandable Explanations for Node Embeddingscited by 2
  6. 2024Appformer: A Novel Framework for Mobile App Usage Prediction Leveraging Progressive Multi-Modal Data Fusion and Feature Extractioncited by 2
  7. 2025Learning Mechanism Underlying NLP Pre-Training and Fine-Tuningcited by 2
  8. 2025Manifold Approximation leads to Robust Kernel Alignmentcited by 2
  9. 2021Effects of Number of Filters of Convolutional Layers on Speech Recognition Model Accuracycited by 1
  10. 2022Micro-level Reserving for General Insurance Claims using a Long Short-Term Memory Networkcited by 1
  11. 2022Statistical process monitoring of artificial neural networkscited by 1
  12. 2023Together We Make Sense -- Learning Meta-Sense Embeddings from Pretrained Static Sense Embeddingscited by 1
  13. 2025Relative Overfitting and Accept-Reject Frameworkcited by 1
  14. 2025Embeddings to Diagnosis: Latent Fragility under Agentic Perturbations in Clinical LLMscited by 1
  15. 2025Factor Augmented Supervised Learning with Text Embeddingscited by 1
  16. 2021SEMIE: SEMantically Infused Embeddings with Enhanced Interpretability for Domain-specific Small Corpuscited by 0
  17. 2023CRL+: A Novel Semi-Supervised Deep Active Contrastive Representation Learning-Based Text Classification Model for Insurance Datacited by 0
  18. 2023Frequency-aware Dimension Selection for Static Word Embedding by Mixed Product Distancecited by 0
  19. 2023Accessing Higher Dimensions for Unsupervised Word Translationcited by 0
  20. 2023An Embedded Diachronic Sense Change Model with a Case Study from Ancient Greekcited by 0
  21. 2023Multi-criteria recommendation systems to foster online grocerycited by 0
  22. 2024Simplicity within biological complexitycited by 0
  23. 2025The cell as a token: high-dimensional geometry in language models and cell embeddingscited by 0
  24. 2025A Topological Improvement of the Overall Performance of Sparse Evolutionary Training: Motif-Based Structural Optimization of Sparse MLPs Projectcited by 0
  25. 2025Knowledge distillation through geometry-aware representational alignmentcited by 0
  26. 2025Higher Embedding Dimension Creates a Stronger World Model for a Simple Sorting Taskcited by 0
  27. 2026Metric k-clustering using only Weak Comparison Oraclescited by 0
  28. 2026ℝ2k is Theoretically Large Enough for Embedding-based Top-k Retrievalcited by 0
  29. 2026Map of Encoders -- Mapping Sentence Encoders using Quantum Relative Entropycited by 0
  30. 2026The Impact of Dimensionality on the Stability of Node Embeddingscited by 0
  31. 2026From Layers to Networks: Comparing Neural Representations via Diffusion Geometrycited by 0
  32. 2026Modular Multimodal Classification Without Fine-Tuning: A Simple Compositional Approachcited by 0
  33. 2026Examining the Limits of Word2Vec with Toki Ponacited by 0
  34. 2026Dynamics of Meaning: Towards the Evaluation of Diachronic Semantic Change in Sinhalacited by 0