mlsanityData to 7 Oct 2026

Obtaining Better Static Word Embeddings Using Contextual Embedding Models

arXiv 2106.04302 ↗First posted 8 Jun 2021cs.CL cs.AI cs.LGcited by 18

Not on the map. Why?

Abstract (arXiv)

The advent of contextual word embeddings -- representations of words which incorporate semantic and syntactic information from their context -- has led to tremendous improvements on a wide variety of NLP tasks. However, recent contextual models have prohibitively high computational cost in many use-cases and are often hard to interpret. In this work, we demonstrate that our proposed distillation method, which is a simple extension of CBOW-based training, allows to significantly improve computational efficiency of NLP applications, while outperforming the quality of existing static embeddings trained from scratch as well as those distilled from previously proposed methods. As a side-effect, our approach also allows a fair comparison of both contextual and static embeddings via standard lexical evaluation tasks.

Key facts. Cited by 18 papers in the map (data through 7 Oct 2026). Builds on 24 of its 57 references that are in the map. First posted on arXiv 8 Jun 2021.

Builds on

24 of 57 references.

References · 24

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  5. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,346
  6. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  7. 2018Deep contextualized word representationscited by 2,285
  8. 2015A large annotated corpus for learning natural language inferencecited by 1,837
  9. 2016Enriching Word Vectors with Subword Informationcited by 1,578
  10. 2019Energy and Policy Considerations for Deep Learning in NLPcited by 1,242
  11. 2019Cross-lingual Language Model Pretrainingcited by 1,088
  12. 2020A Primer in BERTology: What we know about how BERT workscited by 807
12 more
  1. 2019How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddingscited by 770
  2. 2017Word Translation Without Parallel Datacited by 480
  3. 2019Lipstick on a Pig: Debiasing Methods Cover up Systematic Gender Biases in Word Embeddings But do not Remove Themcited by 268
  4. 2014SimLex-999: Evaluating Semantic Models with (Genuine) Similarity Estimationcited by 170
  5. 2017Unsupervised Learning of Sentence Embeddings using Compositional n-Gram Featurescited by 103
  6. 2019Gender-preserving Debiasing for Pre-trained Word Embeddingscited by 61
  7. 2017Learning Joint Multilingual Sentence Representations with Neural Machine Translationcited by 52
  8. 2016SimVerb-3500: A Large-Scale Evaluation Set of Verb Similaritycited by 44
  9. 2020Are All Good Word Vector Spaces Isomorphic?cited by 36
  10. 2016Problems With Evaluation of Word Embeddings Using Word Similarity Taskscited by 25
  11. 2019What do you mean, BERT? Assessing BERT as a Distributional Semantics Modelcited by 14
  12. 2019Better Word Embeddings by Disentangling Contextual n-Gram Informationcited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 5; 2023: 3; 2024: 4; 2025: 3; 2026: 3.

18 papers.

Sort:

By area:

  1. 2023Backpack Language Modelscited by 15
  2. 2022Imputing Out-of-Vocabulary Embeddings with LOVE Makes Language Models Robust with Little Costcited by 9
  3. 2022Combining Static and Contextualised Multilingual Embeddingscited by 8
  4. 2023Distilling Semantic Concept Embeddings from Contrastively Fine-Tuned Language Modelscited by 8
  5. 2022Latent Topology Induction for Understanding Contextualized Representationscited by 7
  6. 2024Extracting Sentence Embeddings from Pretrained Transformer Modelscited by 7
  7. 2024GrEmLIn: A Repository of Green Baseline Embeddings for 87 Low-Resource Languages Injected with Multilingual Graph Knowledgecited by 7
  8. 2022What do Toothbrushes do in the Kitchen? How Transformers Think our World is Structuredcited by 5
  9. 2023A Neighbourhood-Aware Differential Privacy Mechanism for Static Word Embeddingscited by 4
  10. 2024A comparative analysis of embedding models for patent similaritycited by 3
  11. 2024A Comprehensive Analysis of Static Word Embeddings for Turkishcited by 2
  12. 2026Assessing the quality and coherence of word embeddings after SCM-based intersectional bias mitigationcited by 2
6 more
  1. 2022AI Fairness: from Principles to Practicecited by 1
  2. 2025Are Information Retrieval Approaches Good at Harmonising Longitudinal Survey Questions in Social Science?cited by 0
  3. 2025A Comparative Analysis of Static Word Embeddings for Hungariancited by 0
  4. 2025SwiftEmbed: Ultra-Fast Text Embeddings via Static Token Lookup for Real-Time Applicationscited by 0
  5. 2026A Typologically Grounded Evaluation Framework for Word Order and Morphology Sensitivity in Multilingual Masked LMscited by 0
  6. 2026Disentangling Similarity and Relatedness in Topic Modelscited by 0