# TransDrift: Modeling Word-Embedding Drift using Transformer

arXiv: https://arxiv.org/abs/2206.08081 · HTML: https://mlsanity.com/paper/2206.08081/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 36 references that are in the map.
- First posted on arXiv 16 Jun 2022.

## Builds on (most cited first, 16 of 16)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [Enriching Word Vectors with Subword Information](https://mlsanity.com/paper/1607.04606/) (2016)
- [Word Embeddings Quantify 100 Years of Gender and Ethnic Stereotypes](https://mlsanity.com/paper/1711.08412/) (2017)
- [Diachronic Word Embeddings Reveal Statistical Laws of Semantic Change](https://mlsanity.com/paper/1605.09096/) (2016)
- [Diachronic word embeddings and semantic shifts: a survey](https://mlsanity.com/paper/1806.03537/) (2018)
- [Cultural Shift or Linguistic Drift? Comparing Two Computational Measures of Semantic Change](https://mlsanity.com/paper/1606.02821/) (2016)
- [Temporal Analysis of Language through Neural Language Models](https://mlsanity.com/paper/1405.3515/) (2014)
- [Factors Influencing the Surprising Instability of Word Embeddings](https://mlsanity.com/paper/1804.09692/) (2018)
- [Few-Shot Representation Learning for Out-Of-Vocabulary Words](https://mlsanity.com/paper/1907.00505/) (2019)
- [Lifelong Domain Word Embedding via Meta-Learning](https://mlsanity.com/paper/1805.09991/) (2018)
- [Understanding the Downstream Instability of Word Embeddings](https://mlsanity.com/paper/2003.04983/) (2020)
- [Joint Word Representation Learning using a Corpus and a Semantic Lexicon](https://mlsanity.com/paper/1511.06438/) (2015)
- [The Influence of Down-Sampling Strategies on SVD Word Embedding Stability](https://mlsanity.com/paper/1808.06810/) (2018)
- [USFD: Twitter NER with Drift Compensation and Linked Data](https://mlsanity.com/paper/1511.03088/) (2015)
- [Denoising Word Embeddings by Averaging in a Shared Space](https://mlsanity.com/paper/2106.02954/) (2021)

## Built on by (most cited first, 1 of 1)

- [A Theoretical Framework for OOD Robustness in Transformers using Gevrey Classes](https://mlsanity.com/paper/2504.12991/) (2025)

## Abstract (arXiv)

In modern NLP applications, word embeddings are a crucial backbone that can be readily shared across a number of tasks. However as the text distributions change and word semantics evolve over time, the downstream applications using the embeddings can suffer if the word representations do not conform to the data drift. Thus, maintaining word embeddings to be consistent with the underlying data distribution is a key problem. In this work, we tackle this problem and propose TransDrift, a transformer-based prediction model for word embeddings. Leveraging the flexibility of transformer, our model accurately learns the dynamics of the embedding drift and predicts the future embedding. In experiments, we compare with existing methods and show that our model makes significantly more accurate predictions of the word embedding than the baselines. Crucially, by applying the predicted embeddings as a backbone for downstream classification tasks, we show that our embeddings lead to superior performance compared to the previous methods.