Overcoming Poor Word Embeddings with Word Definitions
Not on the map. Why?
Abstract (arXiv)
Modern natural language understanding models depend on pretrained subword embeddings, but applications may need to reason about words that were never or rarely seen during pretraining. We show that examples that depend critically on a rarer word are more challenging for natural language inference models. Then we explore how a model could learn to use definitions, provided in natural text, to overcome this handicap. Our model's understanding of a definition is usually weaker than a well-modeled word embedding, but it recovers most of the performance gap from using a completely untrained word.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 16 of its 19 references that are in the map. First posted on arXiv 5 Mar 2021.
Builds on
16 of 19 references.
References · 16
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
- 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
- 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,375
- 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,811
- 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,989
- 2015A large annotated corpus for learning natural language inferencecited by 1,847
- 2016Enriching Word Vectors with Subword Informationcited by 1,583
- 2016ConceptNet 5.5: An Open Multilingual Graph of General Knowledgecited by 1,067
- 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,043
- 2019Adversarial NLI: A New Benchmark for Natural Language Understandingcited by 807
- 2020Common Sense or World Knowledge? Investigating Adapter-Based Knowledge Injection into Pretrained Transformerscited by 57
- 2020Interpretation of NLP models through input marginalizationcited by 44
4 more
- 2017Neural Natural Language Inference Models Enhanced with External Knowledgecited by 40
- 2020Leap-Of-Thought: Teaching Pre-Trained Models to Systematically Reason Over Implicit Knowledgecited by 24
- 2021Dictionary-based Debiasing of Pre-trained Word Embeddingscited by 24
- 2019BERTRAM: Improved Word Embeddings Have Big Impact on Contextualized Model Performancecited by 19
Built on by
Citing papers per year: 2023: 1.
1 paper.