mlsanityData to 14 Oct 2026

Imputing Out-of-Vocabulary Embeddings with LOVE Makes Language Models Robust with Little Cost

arXiv 2203.07860 ↗First posted 15 Mar 2022cs.CLcited by 9

Not on the map. Why?

Abstract (arXiv)

State-of-the-art NLP systems represent inputs with word embeddings, but these are brittle when faced with Out-of-Vocabulary (OOV) words. To address this issue, we follow the principle of mimick-like models to generate vectors for unseen words, by learning the behavior of pre-trained embeddings using only the surface form of words. We present a simple contrastive learning framework, LOVE, which extends the word representation of an existing pre-trained language model (such as BERT), and makes it robust to OOV with few additional parameters. Extensive evaluations demonstrate that our lightweight model achieves similar or even better performances than prior competitors, both on original datasets and on corrupted variants. Moreover, it can be used in a plug-and-play fashion with FastText and BERT, where it significantly improves their robustness.

Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Builds on 45 of its 70 references that are in the map. First posted on arXiv 15 Mar 2022.

Builds on

45 of 70 references.

References · 45

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  4. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  5. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  6. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  7. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,750
  8. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,375
  9. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,474
  10. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,811
  11. 2018Deep contextualized word representationscited by 2,290
  12. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,173
33 more
  1. 2016Enriching Word Vectors with Subword Informationcited by 1,583
  2. 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,475
  3. 2020Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hyperspherecited by 1,405
  4. 2018Learning deep representations by mutual information estimation and maximizationcited by 1,273
  5. 2019Is BERT Really Robust? A Strong Baseline for Natural Language Attack on Text Classification and Entailmentcited by 671
  6. 2015Bidirectional LSTM-CRF Models for Sequence Taggingcited by 546
  7. 2017Synthetic and Natural Noise Both Break Neural Machine Translationcited by 306
  8. 2020DeCLUTR: Deep Contrastive Learning for Unsupervised Textual Representationscited by 251
  9. 2018An efficient framework for learning sentence representationscited by 198
  10. 2020CLEAR: Contrastive Learning for Sentence Representationcited by 186
  11. 2014SimLex-999: Evaluating Semantic Models with (Genuine) Similarity Estimationcited by 172
  12. 2019Combating Adversarial Misspellings with Robust Word Recognitioncited by 163
  13. 2017Deep Text Classification Can be Fooledcited by 106
  14. 2015A Sensitivity Analysis of (and Practitioners' Guide to) Convolutional Neural Networks for Sentence Classificationcited by 92
  15. 2020CharacterBERT: Reconciling ELMo and BERT for Word-Level Open-Vocabulary Representations From Characterscited by 87
  16. 2017On Sampling Strategies for Neural Network-based Collaborative Filteringcited by 71
  17. 2020Adv-BERT: BERT is not robust on misspellings! Generating nature adversarial samples on BERTcited by 69
  18. 2020CharBERT: Character-aware Pre-trained Language Modelcited by 63
  19. 2017BPEmb: Tokenization-free Pre-trained Subword Embeddings in 275 Languagescited by 52
  20. 2019Rare Words: A Major Problem for Contextualized Embeddings And How to Fix it by Attentive Mimickingcited by 41
  21. 2019Evaluating Word Embedding Models: Methods and Experimental Resultscited by 37
  22. 2017Mimicking Word Embeddings using Subword RNNscited by 30
  23. 2019Few-Shot Representation Learning for Out-Of-Vocabulary Wordscited by 30
  24. 2016Problems With Evaluation of Word Embeddings Using Word Similarity Taskscited by 25
  25. 2019Misspelling Oblivious Word Embeddingscited by 19
  26. 2021Obtaining Better Static Word Embeddings Using Contextual Embedding Modelscited by 18
  27. 2019Attentive Mimicking: Better Word Embeddings by Attending to Informative Contextscited by 17
  28. 2016A Joint Model for Word Embedding and Word Morphologycited by 14
  29. 2016Charagram: Embedding Words and Sentences via Character n-gramscited by 14
  30. 2018Generalizing Word Embeddings using Bag of Subwordscited by 9
  31. 2018Learning Semantic Representations for Novel Words: Leveraging Both Form and Contextcited by 8
  32. 2019A Systematic Study of Leveraging Subword Information for Learning Word Representationscited by 6
  33. 2020PBoS: Probabilistic Bag-of-Subwords for Generalizing Word Embeddingcited by 1

Built on by

Cited by year

Citing papers per year: 2023: 2; 2024: 5; 2025: 2.

9 papers.

  1. 2024Making Them Ask and Answer: Jailbreaking Large Language Models in Few Queries via Disguise and Reconstructioncited by 107
  2. 2024What is the Role of Small Models in the LLM Era: A Surveycited by 41
  3. 2024NEO-BENCH: Evaluating Robustness of Large Language Models with Neologismscited by 20
  4. 2023Vectorizing string entries for data processing on tables: when are larger language models better?cited by 10
  5. 2023Revisit Out-Of-Vocabulary Problem for Slot Filling: A Unified Contrastive Frameword with Multi-level Data Augmentationscited by 8
  6. 2024Learning High-Quality and General-Purpose Phrase Representationscited by 4
  7. 2025Misspellings in Natural Language Processing: A surveycited by 3
  8. 2024Reconsidering Degeneration of Token Embeddings with Definitions for Encoder-based Pre-trained Language Modelscited by 0
  9. 2025Handling Korean Out-of-Vocabulary Words with Phoneme Representation Learningcited by 0