# Dynamic Contextualized Word Embeddings

arXiv: https://arxiv.org/abs/2010.12684 · HTML: https://mlsanity.com/paper/2010.12684/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 Oct 2020.

## Built on by (most cited first, 20 of 20)

- [TimeLMs: Diachronic Language Models from Twitter](https://mlsanity.com/paper/2202.03829/) (2022)
- [Temporal Adaptation of BERT and Performance on Downstream Document Classification: Insights from Social Media](https://mlsanity.com/paper/2104.08116/) (2021)
- [Temporal Effects on Pre-trained Models for Language Processing Tasks](https://mlsanity.com/paper/2111.12790/) (2021)
- [A Survey on Contextualised Semantic Shift Detection](https://mlsanity.com/paper/2304.01666/) (2023)
- [Temporal Attention for Language Models](https://mlsanity.com/paper/2202.02093/) (2022)
- [TempoWiC: An Evaluation Benchmark for Detecting Meaning Shift in Social Media](https://mlsanity.com/paper/2209.07216/) (2022)
- [Geographic Adaptation of Pretrained Language Models](https://mlsanity.com/paper/2203.08565/) (2022)
- [Modeling Ideological Salience and Framing in Polarized Online Groups with Graph Neural Networks and Structured Sparsity](https://mlsanity.com/paper/2104.08829/) (2021)
- [LMSOC: An Approach for Socially Sensitive Pretraining](https://mlsanity.com/paper/2110.10319/) (2021)
- [Time Machine GPT](https://mlsanity.com/paper/2404.18543/) (2024)
- [PerPLM: Personalized Fine-tuning of Pretrained Language Models via Writer-specific Intermediate Learning and Prompts](https://mlsanity.com/paper/2309.07727/) (2023)
- [Evaluating Short-Term Temporal Fluctuations of Social Biases in Social Media Data and Masked Language Models](https://mlsanity.com/paper/2406.13556/) (2024)
- [The challenges of temporal alignment on Twitter during crises](https://mlsanity.com/paper/2104.08535/) (2021)
- [Learning Dynamic Contextualised Word Embeddings via Template-based Temporal Adaptation](https://mlsanity.com/paper/2208.10734/) (2022)
- [Probabilistic Embeddings with Laplacian Graph Priors](https://mlsanity.com/paper/2204.01846/) (2022)
- [Trust in Motion: Capturing Trust Ascendancy in Open-Source Projects using Hybrid AI](https://mlsanity.com/paper/2210.02656/) (2022)
- [Domain-Specific Word Embeddings with Structure Prediction](https://mlsanity.com/paper/2210.04962/) (2022)
- [WIKITIDE: A Wikipedia-Based Timestamped Definition Pairs Dataset](https://mlsanity.com/paper/2308.03582/) (2023)
- [Future Language Modeling from Temporal Document History](https://mlsanity.com/paper/2404.10297/) (2024)
- [Comment-level Topic Drift Analysis in the Reddit Corpus](https://mlsanity.com/paper/2608.19133/) (2026)

## Abstract (arXiv)

Static word embeddings that represent words by a single vector cannot capture the variability of word meaning in different linguistic and extralinguistic contexts. Building on prior work on contextualized and dynamic word embeddings, we introduce dynamic contextualized word embeddings that represent words as a function of both linguistic and extralinguistic context. Based on a pretrained language model (PLM), dynamic contextualized word embeddings model time and social space jointly, which makes them attractive for a range of NLP tasks involving semantic variability. We highlight potential application scenarios by means of qualitative and quantitative analyses on four English datasets.