# Incorporating Word Sense Disambiguation in Neural Language Models

arXiv: https://arxiv.org/abs/2106.07967 · HTML: https://mlsanity.com/paper/2106.07967/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 47 references that are in the map.
- First posted on arXiv 15 Jun 2021.

## Builds on (most cited first, 28 of 28)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://mlsanity.com/paper/1906.08237/) (2019)
- [ALBERT: A Lite BERT for Self-supervised Learning of Language Representations](https://mlsanity.com/paper/1909.11942/) (2019)
- [SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems](https://mlsanity.com/paper/1905.00537/) (2019)
- [Enriching Word Vectors with Subword Information](https://mlsanity.com/paper/1607.04606/) (2016)
- [WiC: the Word-in-Context Dataset for Evaluating Context-Sensitive Meaning Representations](https://mlsanity.com/paper/1808.09121/) (2018)
- [Knowledge Enhanced Contextual Word Representations](https://mlsanity.com/paper/1909.04164/) (2019)
- [GlossBERT: BERT for Word Sense Disambiguation with Gloss Knowledge](https://mlsanity.com/paper/1908.07245/) (2019)
- [SenseBERT: Driving Some Sense into BERT](https://mlsanity.com/paper/1908.05646/) (2019)
- [Moving Down the Long Tail of Word Sense Disambiguation with Gloss-Informed Biencoders](https://mlsanity.com/paper/2005.02590/) (2020)
- [Does BERT Make Any Sense? Interpretable Word Sense Disambiguation with Contextualized Embeddings](https://mlsanity.com/paper/1909.10430/) (2019)
- [Language Modelling Makes Sense: Propagating Representations through WordNet for Full-Coverage Word Sense Disambiguation](https://mlsanity.com/paper/1906.10007/) (2019)
- [Improved Word Sense Disambiguation Using Pre-Trained Contextualized Word Representations](https://mlsanity.com/paper/1910.00194/) (2019)
- [Incorporating Glosses into Neural Word Sense Disambiguation](https://mlsanity.com/paper/1805.08028/) (2018)
- [Are Neural Language Models Good Plagiarists? A Benchmark for Neural Paraphrase Detection](https://mlsanity.com/paper/2103.12450/) (2021)
- [Identifying Machine-Paraphrased Plagiarism](https://mlsanity.com/paper/2103.11909/) (2021)
- [Word Sense Disambiguation using a Bidirectional LSTM](https://mlsanity.com/paper/1606.03568/) (2016)
- [Using BERT for Word Sense Disambiguation](https://mlsanity.com/paper/1909.08358/) (2019)
- [Enhanced word embeddings using multi-semantic representation through lexical chains](https://mlsanity.com/paper/2101.09023/) (2021)
- [Analysis and Evaluation of Language Models for Word Sense Disambiguation](https://mlsanity.com/paper/2008.11608/) (2020)
- [Knowledge-based Word Sense Disambiguation using Topic Models](https://mlsanity.com/paper/1801.01900/) (2018)

## Built on by (most cited first, 6 of 6)

- [A Domain-adaptive Pre-training Approach for Language Bias Detection in News](https://mlsanity.com/paper/2205.10773/) (2022)
- [Specialized Document Embeddings for Aspect-based Similarity of Research Papers](https://mlsanity.com/paper/2203.14541/) (2022)
- [Paraphrase Detection: Human vs. Machine Content](https://mlsanity.com/paper/2303.13989/) (2023)
- [Align before Adapt: Leveraging Entity-to-Region Alignments for Generalizable Video Action Recognition](https://mlsanity.com/paper/2311.15619/) (2023)
- [Vision Meets Definitions: Unsupervised Visual Word Sense Disambiguation Incorporating Gloss Information](https://mlsanity.com/paper/2305.01788/) (2023)
- [Analyzing Multi-Task Learning for Abstractive Text Summarization](https://mlsanity.com/paper/2210.14606/) (2022)

## Abstract (arXiv)

We present two supervised (pre-)training methods to incorporate gloss definitions from lexical resources into neural language models (LMs). The training improves our models' performance for Word Sense Disambiguation (WSD) but also benefits general language understanding tasks while adding almost no parameters. We evaluate our techniques with seven different neural LMs and find that XLNet is more suitable for WSD than BERT. Our best-performing methods exceeds state-of-the-art WSD techniques on the SemCor 3.0 dataset by 0.5% F1 and increase BERT's performance on the GLUE benchmark by 1.1% on average.