# Overcoming Poor Word Embeddings with Word Definitions

arXiv: https://arxiv.org/abs/2103.03842 · HTML: https://mlsanity.com/paper/2103.03842/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 19 references that are in the map.
- First posted on arXiv 5 Mar 2021.

## Builds on (most cited first, 16 of 16)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://mlsanity.com/paper/1906.08237/) (2019)
- [SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing](https://mlsanity.com/paper/1808.06226/) (2018)
- [A large annotated corpus for learning natural language inference](https://mlsanity.com/paper/1508.05326/) (2015)
- [Enriching Word Vectors with Subword Information](https://mlsanity.com/paper/1607.04606/) (2016)
- [ConceptNet 5.5: An Open Multilingual Graph of General Knowledge](https://mlsanity.com/paper/1612.03975/) (2016)
- [Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Books](https://mlsanity.com/paper/1506.06724/) (2015)
- [Adversarial NLI: A New Benchmark for Natural Language Understanding](https://mlsanity.com/paper/1910.14599/) (2019)
- [Common Sense or World Knowledge? Investigating Adapter-Based Knowledge Injection into Pretrained Transformers](https://mlsanity.com/paper/2005.11787/) (2020)
- [Interpretation of NLP models through input marginalization](https://mlsanity.com/paper/2010.13984/) (2020)
- [Neural Natural Language Inference Models Enhanced with External Knowledge](https://mlsanity.com/paper/1711.04289/) (2017)
- [Leap-Of-Thought: Teaching Pre-Trained Models to Systematically Reason Over Implicit Knowledge](https://mlsanity.com/paper/2006.06609/) (2020)
- [Dictionary-based Debiasing of Pre-trained Word Embeddings](https://mlsanity.com/paper/2101.09525/) (2021)
- [BERTRAM: Improved Word Embeddings Have Big Impact on Contextualized Model Performance](https://mlsanity.com/paper/1910.07181/) (2019)

## Built on by (most cited first, 1 of 1)

- [Bilex Rx: Lexical Data Augmentation for Massively Multilingual Machine Translation](https://mlsanity.com/paper/2303.15265/) (2023)

## Abstract (arXiv)

Modern natural language understanding models depend on pretrained subword embeddings, but applications may need to reason about words that were never or rarely seen during pretraining. We show that examples that depend critically on a rarer word are more challenging for natural language inference models. Then we explore how a model could learn to use definitions, provided in natural text, to overcome this handicap. Our model's understanding of a definition is usually weaker than a well-modeled word embedding, but it recovers most of the performance gap from using a completely untrained word.