# Better Word Embeddings by Disentangling Contextual n-Gram Information

arXiv: https://arxiv.org/abs/1904.05033 · HTML: https://mlsanity.com/paper/1904.05033/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 Apr 2019.

## Built on by (most cited first, 12 of 12)

- [On the Transferability of Pre-trained Language Models for Low-Resource Programming Languages](https://mlsanity.com/paper/2204.09653/) (2022)
- [Scaling Embedding Layers in Language Models](https://mlsanity.com/paper/2502.01637/) (2025)
- [Obtaining Better Static Word Embeddings Using Contextual Embedding Models](https://mlsanity.com/paper/2106.04302/) (2021)
- [Enhancing Phenotype Recognition in Clinical Notes Using Large Language Models: PhenoBCBERT and PhenoGPT](https://mlsanity.com/paper/2308.06294/) (2023)
- [Sentence-level Privacy for Document Embeddings](https://mlsanity.com/paper/2205.04605/) (2022)
- [Combating Health Misinformation in Social Media: Characterization, Detection, Intervention, and Open Issues](https://mlsanity.com/paper/2211.05289/) (2022)
- [PAUSE: Positive and Annealed Unlabeled Sentence Embedding](https://mlsanity.com/paper/2109.03155/) (2021)
- [Detecting Anti-Vaccine Users on Twitter](https://mlsanity.com/paper/2110.11333/) (2021)
- [A Collocation-based Method for Addressing Challenges in Word-level Metric Differential Privacy](https://mlsanity.com/paper/2407.00638/) (2024)
- [Zero-Shot Cognitive Impairment Detection from Speech Using AudioLLM](https://mlsanity.com/paper/2506.17351/) (2025)
- [Static Fuzzy Bag-of-Words: a lightweight sentence embedding algorithm](https://mlsanity.com/paper/2304.03098/) (2023)
- [Word Embedding with Neural Probabilistic Prior](https://mlsanity.com/paper/2309.11824/) (2023)

## Abstract (arXiv)

Pre-trained word vectors are ubiquitous in Natural Language Processing applications. In this paper, we show how training word embeddings jointly with bigram and even trigram embeddings, results in improved unigram embeddings. We claim that training word embeddings along with higher n-gram embeddings helps in the removal of the contextual information from the unigrams, resulting in better stand-alone word embeddings. We empirically show the validity of our hypothesis by outperforming other competing word representation models by a significant margin on a wide variety of tasks. We make our models publicly available.