# Word Embedding with Neural Probabilistic Prior

arXiv: https://arxiv.org/abs/2309.11824 · HTML: https://mlsanity.com/paper/2309.11824/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 18 of its 46 references that are in the map.
- First posted on arXiv 21 Sep 2023.

## Builds on (most cited first, 18 of 18)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Semi-Supervised Classification with Graph Convolutional Networks](https://mlsanity.com/paper/1609.02907/) (2016)
- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://mlsanity.com/paper/1906.08237/) (2019)
- [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018)
- [Variational Autoencoders and Nonlinear ICA: A Unifying Framework](https://mlsanity.com/paper/1907.04809/) (2019)
- [SimLex-999: Evaluating Semantic Models with (Genuine) Similarity Estimation](https://mlsanity.com/paper/1408.3456/) (2014)
- [Higher-order Coreference Resolution with Coarse-to-fine Inference](https://mlsanity.com/paper/1804.05392/) (2018)
- [Counter-fitting Word Vectors to Linguistic Constraints](https://mlsanity.com/paper/1603.00892/) (2016)
- [Encoding Sentences with Graph Convolutional Networks for Semantic Role Labeling](https://mlsanity.com/paper/1703.04826/) (2017)
- [Simple and Effective Multi-Paragraph Reading Comprehension](https://mlsanity.com/paper/1710.10723/) (2017)
- [Disentanglement by Nonlinear ICA with General Incompressible-flow Networks (GIN)](https://mlsanity.com/paper/2001.04872/) (2020)
- [Hidden Markov Nonlinear ICA: Unsupervised Learning from Nonstationary Time Series](https://mlsanity.com/paper/2006.12107/) (2020)
- [Better Word Embeddings by Disentangling Contextual n-Gram Information](https://mlsanity.com/paper/1904.05033/) (2019)
- [Embedding Words as Distributions with a Bayesian Skip-gram Model](https://mlsanity.com/paper/1711.11027/) (2017)

## Abstract (arXiv)

To improve word representation learning, we propose a probabilistic prior which can be seamlessly integrated with word embedding models. Different from previous methods, word embedding is taken as a probabilistic generative model, and it enables us to impose a prior regularizing word representation learning. The proposed prior not only enhances the representation of embedding vectors but also improves the model's robustness and stability. The structure of the proposed prior is simple and effective, and it can be easily implemented and flexibly plugged in most existing word embedding models. Extensive experiments show the proposed method improves word representation on various tasks.