mlsanityData to 7 Oct 2026

Using Context-to-Vector with Graph Retrofitting to Improve Word Embeddings

arXiv 2210.16848 ↗First posted 30 Oct 2022cs.CL cs.AIcited by 11

Not on the map. Why?

Abstract (arXiv)

Although contextualized embeddings generated from large-scale pre-trained models perform well in many tasks, traditional static embeddings (e.g., Skip-gram, Word2Vec) still play an important role in low-resource and lightweight settings due to their low computational cost, ease of deployment, and stability. In this paper, we aim to improve word embeddings by 1) incorporating more contextual information from existing pre-trained models into the Skip-gram framework, which we call Context-to-Vec; 2) proposing a post-processing retrofitting method for static embeddings independent of training by employing priori synonym knowledge and weighted vector distribution. Through extrinsic and intrinsic tasks, our methods are well proven to outperform the baselines by a large margin.

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Builds on 12 of its 33 references that are in the map. First posted on arXiv 30 Oct 2022.

Builds on

12 of 33 references.

References · 12

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,346
  5. 2018UMAP: Uniform Manifold Approximation and Projection for Dimension Reductioncited by 2,841
  6. 2018Deep contextualized word representationscited by 2,285
  7. 2016Enriching Word Vectors with Subword Informationcited by 1,578
  8. 2021Self-supervised Learning on Graphs: Contrastive, Generative,or Predictivecited by 136
  9. 2018NCRF++: An Open-source Neural Sequence Labeling Toolkitcited by 29
  10. 2022A Survey of Pretraining on Graphs: Taxonomy, Methods, and Applicationscited by 23
  11. 2020Contextual Embeddings: When Are They Worth It?cited by 15
  12. 2018Uncovering divergent linguistic information in word embeddings with lessons for intrinsic and extrinsic evaluationcited by 8

Built on by

Cited by year

Citing papers per year: 2022: 1; 2023: 4; 2024: 5; 2025: 1.

11 papers.

  1. 2024A Survey on Multilingual Large Language Models: Corpora, Alignment, and Biascited by 82
  2. 2023CVT-SLR: Contrastive Visual-Textual Transformation for Sign Language Recognition with Variational Alignmentcited by 44
  3. 2024Extracting Sentence Embeddings from Pretrained Transformer Modelscited by 7
  4. 2023Progressive Multi-Modality Learning for Inverse Protein Foldingcited by 5
  5. 2022Leveraging Graph-based Cross-modal Information Fusion for Neural Sign Language Translationcited by 4
  6. 2023CCPL: Cross-modal Contrastive Protein Learningcited by 4
  7. 2023Why Deep Models Often cannot Beat Non-deep Counterparts on Molecular Property Prediction?cited by 3
  8. 2024MetaEnzyme: Meta Pan-Enzyme Learning for Task-Adaptive Redesigncited by 2
  9. 2024Pan-protein Design Learning Enables Task-adaptive Generalization for Low-resource Enzyme Designcited by 0
  10. 2024DapPep: Domain Adaptive Peptide-agnostic Learning for Universal T-cell Receptor-antigen Binding Affinity Predictioncited by 0
  11. 2025Domain Lexical Knowledge-based Word Embedding Learning for Text Classification under Small Datacited by 0