# Lacking the embedding of a word? Look it up into a traditional dictionary

arXiv: https://arxiv.org/abs/2109.11763 · HTML: https://mlsanity.com/paper/2109.11763/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 71 references that are in the map.
- First posted on arXiv 24 Sep 2021.

## Builds on (most cited first, 28 of 28)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://mlsanity.com/paper/1906.08237/) (2019)
- [DeBERTa: Decoding-enhanced BERT with Disentangled Attention](https://mlsanity.com/paper/2006.03654/) (2020)
- [Enriching Word Vectors with Subword Information](https://mlsanity.com/paper/1607.04606/) (2016)
- [Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation](https://mlsanity.com/paper/1609.08144/) (2016)
- [Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Books](https://mlsanity.com/paper/1506.06724/) (2015)
- [Bag of Tricks for Efficient Text Classification](https://mlsanity.com/paper/1607.01759/) (2016)
- [Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference](https://mlsanity.com/paper/1902.01007/) (2019)
- [What you can cram into a single vector: Probing sentence embeddings for linguistic properties](https://mlsanity.com/paper/1805.01070/) (2018)
- [Improved Semantic Representations From Tree-Structured Long Short-Term Memory Networks](https://mlsanity.com/paper/1503.00075/) (2015)
- [Learning Word Vectors for 157 Languages](https://mlsanity.com/paper/1802.06893/) (2018)
- [TOD-BERT: Pre-trained Natural Language Understanding for Task-Oriented Dialogue](https://mlsanity.com/paper/2004.06871/) (2020)
- [SimLex-999: Evaluating Semantic Models with (Genuine) Similarity Estimation](https://mlsanity.com/paper/1408.3456/) (2014)
- [A Convolutional Neural Network for Modelling Sentences](https://mlsanity.com/paper/1404.2188/) (2014)
- [DocBERT: BERT for Document Classification](https://mlsanity.com/paper/1904.08398/) (2019)
- [SimVerb-3500: A Large-Scale Evaluation Set of Verb Similarity](https://mlsanity.com/paper/1608.00869/) (2016)
- [Learning to Understand Phrases by Embedding the Dictionary](https://mlsanity.com/paper/1504.00548/) (2015)
- [Few-Shot Representation Learning for Out-Of-Vocabulary Words](https://mlsanity.com/paper/1907.00505/) (2019)
- [Linguistic Profiling of a Neural Language Model](https://mlsanity.com/paper/2010.01869/) (2020)
- [BERTRAM: Improved Word Embeddings Have Big Impact on Contextualized Model Performance](https://mlsanity.com/paper/1910.07181/) (2019)
- [A La Carte Embedding: Cheap but Effective Induction of Semantic Feature Vectors](https://mlsanity.com/paper/1805.05388/) (2018)
- [Attentive Mimicking: Better Word Embeddings by Attending to Informative Contexts](https://mlsanity.com/paper/1904.01617/) (2019)
- [High-risk learning: acquiring new word vectors from tiny data](https://mlsanity.com/paper/1707.06556/) (2017)
- [Learning to Compute Word Embeddings On the Fly](https://mlsanity.com/paper/1706.00286/) (2017)
- [Symbolic, Distributed and Distributional Representations for Natural Language Processing in the Era of Deep Learning: a Survey](https://mlsanity.com/paper/1702.00764/) (2017)
- [Unseen Word Representation by Aligning Heterogeneous Lexical Semantic Spaces](https://mlsanity.com/paper/1811.04983/) (2018)

## Built on by (most cited first, 4 of 4)

- [Dict-BERT: Enhancing Language Model Pre-training with Dictionary](https://mlsanity.com/paper/2110.06490/) (2021)
- [Injecting knowledge into language generation: a case study in auto-charting after-visit care instructions from medical dialogue](https://mlsanity.com/paper/2306.03652/) (2023)
- [Reconsidering Degeneration of Token Embeddings with Definitions for Encoder-based Pre-trained Language Models](https://mlsanity.com/paper/2408.01308/) (2024)
- [A Similarity-Based Oversampling Method for Multi-label Imbalanced Text Data](https://mlsanity.com/paper/2411.01013/) (2024)

## Abstract (arXiv)

Word embeddings are powerful dictionaries, which may easily capture language variations. However, these dictionaries fail to give sense to rare words, which are surprisingly often covered by traditional dictionaries. In this paper, we propose to use definitions retrieved in traditional dictionaries to produce word embeddings for rare words. For this purpose, we introduce two methods: Definition Neural Network (DefiNNet) and Define BERT (DefBERT). In our experiments, DefiNNet and DefBERT significantly outperform state-of-the-art as well as baseline methods devised for producing embeddings of unknown words. In fact, DefiNNet significantly outperforms FastText, which implements a method for the same task-based on n-grams, and DefBERT significantly outperforms the BERT method for OOV words. Then, definitions in traditional dictionaries are useful to build word embeddings for rare words.