# Dynamic Word Embeddings

arXiv: https://arxiv.org/abs/1702.08359 · HTML: https://mlsanity.com/paper/1702.08359/

## Key facts

- Cited by 51 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 27 Feb 2017.

## Built on by (most cited first, 30 of 51)

- [Time-Aware Language Models as Temporal Knowledge Bases](https://mlsanity.com/paper/2106.15110/) (2021)
- [A Survey on Embedding Dynamic Graphs](https://mlsanity.com/paper/2101.01229/) (2021)
- [Entity Cloze By Date: What LMs Know About Unseen Entities](https://mlsanity.com/paper/2205.02832/) (2022)
- [Time Masking for Temporal Language Models](https://mlsanity.com/paper/2110.06366/) (2021)
- [Time is Encoded in the Weights of Finetuned Language Models](https://mlsanity.com/paper/2312.13401/) (2023)
- [Dynamic Language Models for Continuously Evolving Content](https://mlsanity.com/paper/2106.06297/) (2021)
- [Simple, Interpretable and Stable Method for Detecting Words with Usage Change across Corpora](https://mlsanity.com/paper/2112.14330/) (2021)
- [Temporal Attention for Language Models](https://mlsanity.com/paper/2202.02093/) (2022)
- [Improving Temporal Generalization of Pre-trained Language Models with Lexical Semantic Change](https://mlsanity.com/paper/2210.17127/) (2022)
- [Challenges for Computational Lexical Semantic Change](https://mlsanity.com/paper/2101.07668/) (2021)
- [Markovian Gaussian Process Variational Autoencoders](https://mlsanity.com/paper/2207.05543/) (2022)
- [Understanding Entropy Coding With Asymmetric Numeral Systems (ANS): a Statistician's Perspective](https://mlsanity.com/paper/2201.01741/) (2022)
- [HistBERT: A Pre-trained Language Model for Diachronic Lexical Semantic Analysis](https://mlsanity.com/paper/2202.03612/) (2022)
- [Once Upon a textitTime in textitGraph: Relative-Time Pretraining for Complex Temporal Reasoning](https://mlsanity.com/paper/2310.14709/) (2023)
- [Neural Dynamic Focused Topic Model](https://mlsanity.com/paper/2301.10988/) (2023)
- [On Disentanglement in Gaussian Process Variational Autoencoders](https://mlsanity.com/paper/2102.05507/) (2021)
- [Theoretical foundations and limits of word embeddings: what types of meaning can they capture?](https://mlsanity.com/paper/2107.10413/) (2021)
- [TempoFormer: A Transformer for Temporally-aware Representations in Change Detection](https://mlsanity.com/paper/2408.15689/) (2024)
- [A diachronic evaluation of gender asymmetry in euphemism](https://mlsanity.com/paper/2106.02083/) (2021)
- [On the Universality of Deep Contextual Language Models](https://mlsanity.com/paper/2109.07140/) (2021)
- [adSformers: Personalization from Short-Term Sequences and Diversity of Representations in Etsy Ads](https://mlsanity.com/paper/2302.01255/) (2023)
- [Letters From the Past: Modeling Historical Sound Change Through Diachronic Character Embeddings](https://mlsanity.com/paper/2205.08256/) (2022)
- [Syntactic Language Change in English and German: Metrics, Parsers, and Convergences](https://mlsanity.com/paper/2402.11549/) (2024)
- [Discovering emergent connections in quantum physics research via dynamic word embeddings](https://mlsanity.com/paper/2411.06577/) (2024)
- [The challenges of temporal alignment on Twitter during crises](https://mlsanity.com/paper/2104.08535/) (2021)
- [Sig-Networks Toolkit: Signature Networks for Longitudinal Language Modelling](https://mlsanity.com/paper/2312.03523/) (2023)
- [Fake it Till You Make it: Self-Supervised Semantic Shifts for Monolingual Word Embedding Tasks](https://mlsanity.com/paper/2102.00290/) (2021)
- [Measuring diachronic sense change: new models and Monte Carlo methods for Bayesian inference](https://mlsanity.com/paper/2105.00819/) (2021)
- [Probabilistic Embeddings with Laplacian Graph Priors](https://mlsanity.com/paper/2204.01846/) (2022)
- [Metadata Might Make Language Models Better](https://mlsanity.com/paper/2211.10086/) (2022)

## Abstract (arXiv)

We present a probabilistic language model for time-stamped text data which tracks the semantic evolution of individual words over time. The model represents words and contexts by latent trajectories in an embedding space. At each moment in time, the embedding vectors are inferred from a probabilistic version of word2vec [Mikolov et al., 2013]. These embedding vectors are connected in time through a latent diffusion process. We describe two scalable variational inference algorithms--skip-gram smoothing and skip-gram filtering--that allow us to train the model jointly over all times; thus learning on all data while simultaneously allowing word and context vectors to drift. Experimental results on three different corpora demonstrate that our dynamic model infers word embedding trajectories that are more interpretable and lead to higher predictive likelihoods than competing methods that are based on static models trained separately on time slices.