# Using Context-to-Vector with Graph Retrofitting to Improve Word Embeddings

arXiv: https://arxiv.org/abs/2210.16848 · HTML: https://mlsanity.com/paper/2210.16848/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 33 references that are in the map.
- First posted on arXiv 30 Oct 2022.

## Builds on (most cited first, 12 of 12)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction](https://mlsanity.com/paper/1802.03426/) (2018)
- [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018)
- [Enriching Word Vectors with Subword Information](https://mlsanity.com/paper/1607.04606/) (2016)
- [Self-supervised Learning on Graphs: Contrastive, Generative,or Predictive](https://mlsanity.com/paper/2105.07342/) (2021)
- [NCRF++: An Open-source Neural Sequence Labeling Toolkit](https://mlsanity.com/paper/1806.05626/) (2018)
- [A Survey of Pretraining on Graphs: Taxonomy, Methods, and Applications](https://mlsanity.com/paper/2202.07893/) (2022)
- [Contextual Embeddings: When Are They Worth It?](https://mlsanity.com/paper/2005.09117/) (2020)
- [Uncovering divergent linguistic information in word embeddings with lessons for intrinsic and extrinsic evaluation](https://mlsanity.com/paper/1809.02094/) (2018)

## Built on by (most cited first, 11 of 11)

- [A Survey on Multilingual Large Language Models: Corpora, Alignment, and Bias](https://mlsanity.com/paper/2404.00929/) (2024)
- [CVT-SLR: Contrastive Visual-Textual Transformation for Sign Language Recognition with Variational Alignment](https://mlsanity.com/paper/2303.05725/) (2023)
- [Extracting Sentence Embeddings from Pretrained Transformer Models](https://mlsanity.com/paper/2408.08073/) (2024)
- [Progressive Multi-Modality Learning for Inverse Protein Folding](https://mlsanity.com/paper/2312.06297/) (2023)
- [Leveraging Graph-based Cross-modal Information Fusion for Neural Sign Language Translation](https://mlsanity.com/paper/2211.00526/) (2022)
- [CCPL: Cross-modal Contrastive Protein Learning](https://mlsanity.com/paper/2303.11783/) (2023)
- [Why Deep Models Often cannot Beat Non-deep Counterparts on Molecular Property Prediction?](https://mlsanity.com/paper/2306.17702/) (2023)
- [MetaEnzyme: Meta Pan-Enzyme Learning for Task-Adaptive Redesign](https://mlsanity.com/paper/2408.10247/) (2024)
- [Pan-protein Design Learning Enables Task-adaptive Generalization for Low-resource Enzyme Design](https://mlsanity.com/paper/2411.17795/) (2024)
- [DapPep: Domain Adaptive Peptide-agnostic Learning for Universal T-cell Receptor-antigen Binding Affinity Prediction](https://mlsanity.com/paper/2411.17798/) (2024)
- [Domain Lexical Knowledge-based Word Embedding Learning for Text Classification under Small Data](https://mlsanity.com/paper/2506.01621/) (2025)

## Abstract (arXiv)

Although contextualized embeddings generated from large-scale pre-trained models perform well in many tasks, traditional static embeddings (e.g., Skip-gram, Word2Vec) still play an important role in low-resource and lightweight settings due to their low computational cost, ease of deployment, and stability. In this paper, we aim to improve word embeddings by 1) incorporating more contextual information from existing pre-trained models into the Skip-gram framework, which we call Context-to-Vec; 2) proposing a post-processing retrofitting method for static embeddings independent of training by employing priori synonym knowledge and weighted vector distribution. Through extrinsic and intrinsic tasks, our methods are well proven to outperform the baselines by a large margin.