# word2vec Parameter Learning Explained

arXiv: https://arxiv.org/abs/1411.2738 · HTML: https://mlsanity.com/paper/1411.2738/

## Key facts

- Cited by 53 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 Nov 2014.

## Built on by (most cited first, 30 of 53)

- [Graph Learning: A Survey](https://mlsanity.com/paper/2105.00696/) (2021)
- [Membership Inference on Word Embedding and Beyond](https://mlsanity.com/paper/2106.11384/) (2021)
- [Trustworthy Recommender Systems](https://mlsanity.com/paper/2208.06265/) (2022)
- [Recent Developments in Recommender Systems: A Survey](https://mlsanity.com/paper/2306.12680/) (2023)
- [RPT: Toward Transferable Model on Heterogeneous Researcher Data via Pre-Training](https://mlsanity.com/paper/2110.07336/) (2021)
- [Omniview-Tuning: Boosting Viewpoint Invariance of Vision-Language Pre-training Models](https://mlsanity.com/paper/2404.12139/) (2024)
- [A Mechanism for Producing Aligned Latent Spaces with Autoencoders](https://mlsanity.com/paper/2106.15456/) (2021)
- [Theoretical foundations and limits of word embeddings: what types of meaning can they capture?](https://mlsanity.com/paper/2107.10413/) (2021)
- [Estimating Predictive Uncertainty Under Program Data Distribution Shift](https://mlsanity.com/paper/2107.10989/) (2021)
- [Semantic-Preserving Adversarial Text Attacks](https://mlsanity.com/paper/2108.10015/) (2021)
- [A Versatile Influence Function for Data Attribution with Non-Decomposable Loss](https://mlsanity.com/paper/2412.01335/) (2024)
- [Dimension Agnostic Neural Processes](https://mlsanity.com/paper/2502.20661/) (2025)
- [quantum Case-Based Reasoning (qCBR)](https://mlsanity.com/paper/2104.00409/) (2021)
- [Deep learning-based NLP Data Pipeline for EHR Scanned Document Information Extraction](https://mlsanity.com/paper/2110.11864/) (2021)
- [Generating Post-hoc Explanations for Skip-gram-based Node Embeddings by Identifying Important Nodes with Bridgeness](https://mlsanity.com/paper/2304.12036/) (2023)
- [UniTE: A Survey and Unified Pipeline for Pre-training Spatiotemporal Trajectory Embeddings](https://mlsanity.com/paper/2407.12550/) (2024)
- [Are LLM-Enhanced Graph Neural Networks Robust against Poisoning Attacks?](https://mlsanity.com/paper/2603.26105/) (2026)
- [Network Representation Learning: From Traditional Feature Learning to Deep Learning](https://mlsanity.com/paper/2103.04339/) (2021)
- [Word Embedding for Social Sciences: An Interdisciplinary Survey](https://mlsanity.com/paper/2207.03086/) (2022)
- [Do Large GPT Models Discover Moral Dimensions in Language Representations? A Topological Study Of Sentence Embeddings](https://mlsanity.com/paper/2309.09397/) (2023)
- [Revolutionizing Bridge Operation and Maintenance with LLM-based Agents: An Overview of Applications and Insights](https://mlsanity.com/paper/2407.10064/) (2024)
- [The Golden Rule as a Heuristic to Measure the Fairness of Texts Using Machine Learning](https://mlsanity.com/paper/2111.00107/) (2021)
- [Effects of Number of Filters of Convolutional Layers on Speech Recognition Model Accuracy](https://mlsanity.com/paper/2102.02326/) (2021)
- [DCDetector: An IoT terminal vulnerability mining system based on distributed deep ensemble learning under source code representation](https://mlsanity.com/paper/2211.16235/) (2022)
- [Neural Controller Synthesis for Signal Temporal Logic Specifications Using Encoder-Decoder Structured Networks](https://mlsanity.com/paper/2212.05200/) (2022)
- [LASIGE and UNICAGE solution to the NASA LitCoin NLP Competition](https://mlsanity.com/paper/2308.05609/) (2023)
- [Multiscale Hodge Scattering Networks for Data Analysis](https://mlsanity.com/paper/2311.10270/) (2023)
- [Understanding the Progression of Educational Topics via Semantic Matching](https://mlsanity.com/paper/2403.05553/) (2024)
- [Correspondence Analysis and PMI-Based Word Embeddings: A Comparative Study](https://mlsanity.com/paper/2405.20895/) (2024)
- [Enhancing Interpretability in Deep Reinforcement Learning through Semantic Clustering](https://mlsanity.com/paper/2409.17411/) (2024)

## Abstract (arXiv)

The word2vec model and application by Mikolov et al. have attracted a great amount of attention in recent two years. The vector representations of words learned by word2vec models have been shown to carry semantic meanings and are useful in various NLP tasks. As an increasing number of researchers would like to experiment with word2vec or similar techniques, I notice that there lacks a material that comprehensively explains the parameter learning process of word embedding models in details, thus preventing researchers that are non-experts in neural networks from understanding the working mechanism of such models. This note provides detailed derivations and explanations of the parameter update equations of the word2vec models, including the original continuous bag-of-word (CBOW) and skip-gram (SG) models, as well as advanced optimization techniques, including hierarchical softmax and negative sampling. Intuitive interpretations of the gradient equations are also provided alongside mathematical derivations. In the appendix, a review on the basics of neuron networks and backpropagation is provided. I also created an interactive demo, wevi, to facilitate the intuitive understanding of the model.