mlsanityData to 7 Oct 2026

Incorporating Word Sense Disambiguation in Neural Language Models

arXiv 2106.07967 ↗First posted 15 Jun 2021cs.CL cs.AIcited by 6

Not on the map. Why?

Abstract (arXiv)

We present two supervised (pre-)training methods to incorporate gloss definitions from lexical resources into neural language models (LMs). The training improves our models' performance for Word Sense Disambiguation (WSD) but also benefits general language understanding tasks while adding almost no parameters. We evaluate our techniques with seven different neural LMs and find that XLNet is more suitable for WSD than BERT. Our best-performing methods exceeds state-of-the-art WSD techniques on the SemCor 3.0 dataset by 0.5% F1 and increase BERT's performance on the GLUE benchmark by 1.1% on average.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 28 of its 47 references that are in the map. First posted on arXiv 15 Jun 2021.

Builds on

28 of 47 references.

References · 28

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  5. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  6. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,346
  7. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  8. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,462
  9. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  10. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  11. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  12. 2016Enriching Word Vectors with Subword Informationcited by 1,578
16 more
  1. 2018WiC: the Word-in-Context Dataset for Evaluating Context-Sensitive Meaning Representationscited by 370
  2. 2019Knowledge Enhanced Contextual Word Representationscited by 294
  3. 2019GlossBERT: BERT for Word Sense Disambiguation with Gloss Knowledgecited by 78
  4. 2019SenseBERT: Driving Some Sense into BERTcited by 73
  5. 2020Moving Down the Long Tail of Word Sense Disambiguation with Gloss-Informed Biencoderscited by 61
  6. 2019Does BERT Make Any Sense? Interpretable Word Sense Disambiguation with Contextualized Embeddingscited by 55
  7. 2019Language Modelling Makes Sense: Propagating Representations through WordNet for Full-Coverage Word Sense Disambiguationcited by 40
  8. 2019Improved Word Sense Disambiguation Using Pre-Trained Contextualized Word Representationscited by 21
  9. 2018Incorporating Glosses into Neural Word Sense Disambiguationcited by 18
  10. 2021Are Neural Language Models Good Plagiarists? A Benchmark for Neural Paraphrase Detectioncited by 17
  11. 2021Identifying Machine-Paraphrased Plagiarismcited by 16
  12. 2016Word Sense Disambiguation using a Bidirectional LSTMcited by 11
  13. 2019Using BERT for Word Sense Disambiguationcited by 7
  14. 2021Enhanced word embeddings using multi-semantic representation through lexical chainscited by 7
  15. 2020Analysis and Evaluation of Language Models for Word Sense Disambiguationcited by 6
  16. 2018Knowledge-based Word Sense Disambiguation using Topic Modelscited by 5

Built on by

Cited by year

Citing papers per year: 2022: 3; 2023: 3.

6 papers.

  1. 2022A Domain-adaptive Pre-training Approach for Language Bias Detection in Newscited by 16
  2. 2022Specialized Document Embeddings for Aspect-based Similarity of Research Paperscited by 12
  3. 2023Paraphrase Detection: Human vs. Machine Contentcited by 12
  4. 2023Align before Adapt: Leveraging Entity-to-Region Alignments for Generalizable Video Action Recognitioncited by 12
  5. 2023Vision Meets Definitions: Unsupervised Visual Word Sense Disambiguation Incorporating Gloss Informationcited by 5
  6. 2022Analyzing Multi-Task Learning for Abstractive Text Summarizationcited by 3