# Domain-Specific Word Embeddings with Structure Prediction

arXiv: https://arxiv.org/abs/2210.04962 · HTML: https://mlsanity.com/paper/2210.04962/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 16 of its 44 references that are in the map.
- First posted on arXiv 6 Oct 2022.

## Builds on (most cited first, 16 of 16)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [Bag of Tricks for Efficient Text Classification](https://mlsanity.com/paper/1607.01759/) (2016)
- [Diachronic Word Embeddings Reveal Statistical Laws of Semantic Change](https://mlsanity.com/paper/1605.09096/) (2016)
- [Exploiting Similarities among Languages for Machine Translation](https://mlsanity.com/paper/1309.4168/) (2013)
- [Unsupervised Alignment of Embeddings with Wasserstein Procrustes](https://mlsanity.com/paper/1805.11222/) (2018)
- [Diachronic word embeddings and semantic shifts: a survey](https://mlsanity.com/paper/1806.03537/) (2018)
- [Statistically Significant Detection of Linguistic Change](https://mlsanity.com/paper/1411.3315/) (2014)
- [Survey of Computational Approaches to Lexical Semantic Change](https://mlsanity.com/paper/1811.06278/) (2018)
- [Temporal Analysis of Language through Neural Language Models](https://mlsanity.com/paper/1405.3515/) (2014)
- [Dynamic Word Embeddings](https://mlsanity.com/paper/1702.08359/) (2017)
- [Dynamic Word Embeddings for Evolving Semantic Discovery](https://mlsanity.com/paper/1703.00607/) (2017)
- [Simple, Interpretable and Stable Method for Detecting Words with Usage Change across Corpora](https://mlsanity.com/paper/2112.14330/) (2021)
- [Dynamic Contextualized Word Embeddings](https://mlsanity.com/paper/2010.12684/) (2020)
- [Exponential Family Embeddings](https://mlsanity.com/paper/1608.00778/) (2016)

## Abstract (arXiv)

Complementary to finding good general word embeddings, an important question for representation learning is to find dynamic word embeddings, e.g., across time or domain. Current methods do not offer a way to use or predict information on structure between sub-corpora, time or domain and dynamic embeddings can only be compared after post-alignment. We propose novel word embedding methods that provide general word representations for the whole corpus, domain-specific representations for each sub-corpus, sub-corpus structure, and embedding alignment simultaneously. We present an empirical evaluation on New York Times articles and two English Wikipedia datasets with articles on science and philosophy. Our method, called Word2Vec with Structure Prediction (W2VPred), provides better performance than baselines in terms of the general analogy tests, domain-specific analogy tests, and multiple specific word embedding evaluations as well as structure prediction performance when no structure is given a priori. As a use case in the field of Digital Humanities we demonstrate how to raise novel research questions for high literature from the German Text Archive.