# An Exploration of Word Embedding Initialization in Deep-Learning Tasks

arXiv: https://arxiv.org/abs/1711.09160 · HTML: https://mlsanity.com/paper/1711.09160/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 Nov 2017.

## Built on by (most cited first, 6 of 6)

- [Universal Multimodal Representation for Language Understanding](https://mlsanity.com/paper/2301.03344/) (2023)
- [Snowflake: Scaling GNNs to High-Dimensional Continuous Control via Parameter Freezing](https://mlsanity.com/paper/2103.01009/) (2021)
- [Knowledge Graph Embedding Methods for Entity Alignment: An Experimental Review](https://mlsanity.com/paper/2203.09280/) (2022)
- [Improving Data Driven Inverse Text Normalization using Data Augmentation](https://mlsanity.com/paper/2207.09674/) (2022)
- [On Initializing Transformers with Pre-trained Embeddings](https://mlsanity.com/paper/2407.12514/) (2024)
- [Let Triggers Control: Frequency-Aware Dropout for Effective Token Control](https://mlsanity.com/paper/2603.27199/) (2026)

## Abstract (arXiv)

Word embeddings are the interface between the world of discrete units of text processing and the continuous, differentiable world of neural networks. In this work, we examine various random and pretrained initialization methods for embeddings used in deep networks and their effect on the performance on four NLP tasks with both recurrent and convolutional architectures. We confirm that pretrained embeddings are a little better than random initialization, especially considering the speed of learning. On the other hand, we do not see any significant difference between various methods of random initialization, as long as the variance is kept reasonably low. High-variance initialization prevents the network to use the space of embeddings and forces it to use other free parameters to accomplish the task. We support this hypothesis by observing the performance in learning lexical relations and by the fact that the network can learn to perform reasonably in its task even with fixed random embeddings.