mlsanityData to 7 Oct 2026

SenseBERT: Driving Some Sense into BERT

arXiv 1908.05646 ↗First posted 15 Aug 2019cs.CL cs.LGcited by 73

Not on the map. Why?

Abstract (arXiv)

The ability to learn from large unlabeled corpora has allowed neural language models to advance the frontier in natural language understanding. However, existing self-supervision techniques operate at the word form level, which serves as a surrogate for the underlying semantic content. This paper proposes a method to employ weak-supervision directly at the word sense level. Our model, named SenseBERT, is pre-trained to predict not only the masked words but also their WordNet supersenses. Accordingly, we attain a lexical-semantic level language model, without the use of human annotation. SenseBERT achieves significantly improved lexical understanding, as we demonstrate by experimenting on SemEval Word Sense Disambiguation, and by attaining a state of the art result on the Word in Context task.

Key facts. Cited by 73 papers in the map (data through 7 Oct 2026). First posted on arXiv 15 Aug 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 26; 2022: 18; 2023: 15; 2024: 6; 2025: 4; 2026: 4.

73 papers.

Sort:

By area:

  1. 2021AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processingcited by 131
  2. 2022A Survey of Knowledge Enhanced Pre-trained Language Modelscited by 76
  3. 2023Give Us the Facts: Enhancing Large Language Models with Knowledge Graphs for Fact-aware Language Modelingcited by 56
  4. 2023Mixture-of-Domain-Adapters: Decoupling and Injecting Domain Knowledge to Pre-trained Language Models Memoriescited by 34
  5. 2021Knowledge Enhanced Pretrained Language Models: A Compreshensive Surveycited by 33
  6. 2021Learning Domain-Specialised Representations for Cross-Lingual Biomedical Entity Linkingcited by 27
  7. 2021Reasoning over Vision and Language: Exploring the Benefits of Supplemental Knowledgecited by 24
  8. 2021NeuralLog: Natural Language Inference with Joint Neural and Logical Reasoningcited by 24
  9. 2022Knowledge-in-Context: Towards Knowledgeable Semi-Parametric Language Modelscited by 24
  10. 2023The Life Cycle of Knowledge in Big Language Models: A Surveycited by 23
  11. 2021MirrorWiC: On Eliciting Word-in-Context Representations from Pretrained Language Modelscited by 18
  12. 2022Sense Embeddings are also Biased--Evaluating Social Biases in Static and Contextualised Sense Embeddingscited by 18
38 more
  1. 2021LMMS Reloaded: Transformer-based Sense Embeddings for Disambiguation and Beyondcited by 17
  2. 2022InforMask: Unsupervised Informative Masking for Language Model Pretrainingcited by 15
  3. 2021Knowledge Enhanced Fine-Tuning for Better Handling Unseen Entities in Dialogue Generationcited by 13
  4. 2023Driving Context into Text-to-Text Privatizationcited by 13
  5. 2021K-PLUG: Knowledge-injected Pre-trained Language Model for Natural Language Understanding and Generation in E-Commercecited by 12
  6. 2021CausalBERT: Injecting Causal Knowledge Into Pre-trained Models with Minimal Supervisioncited by 12
  7. 2022Knowledge Infused Decodingcited by 12
  8. 2021KI-BERT: Infusing Knowledge Context for Better Language and Domain Understandingcited by 11
  9. 2023Crosslingual Transfer Learning for Low-Resource Languages Based on Multilingual Colexification Graphscited by 11
  10. 2021RAW-C: Relatedness of Ambiguous Words--in Context (A New Lexical Resource for English)cited by 10
  11. 2021Using Knowledge-Embedded Attention to Augment Pre-trained Language Models for Fine-Grained Emotion Recognitioncited by 10
  12. 2021Uniform Manifold Approximation and Projection (UMAP) and its Variants: Tutorial and Surveycited by 10
  13. 2024KnowLA: Enhancing Parameter-efficient Finetuning with Knowledgeable Adaptationcited by 10
  14. 2021Mixed Attention Transformer for Leveraging Word-Level Knowledge to Neural Cross-Lingual Information Retrievalcited by 8
  15. 2022SLISEMAP: Supervised dimensionality reduction through local explanationscited by 7
  16. 2024Analyzing Semantic Change through Lexical Replacementscited by 7
  17. 2021Incorporating Word Sense Disambiguation in Neural Language Modelscited by 6
  18. 2021LexSubCon: Integrating Knowledge from Lexical Resources into Contextual Embeddings for Lexical Substitutioncited by 6
  19. 2021Learning Sense-Specific Static Embeddings using Contextualised Word Embeddings as a Proxycited by 6
  20. 2023Plug-and-Play Knowledge Injection for Pre-trained Language Modelscited by 6
  21. 2023Towards Concept-Aware Large Language Modelscited by 6
  22. 2021Connect-the-Dots: Bridging Semantics between Words and Definitions via Aligning Word Sense Inventoriescited by 5
  23. 2022Better Language Model with Hypernym Class Predictioncited by 5
  24. 2023Bridging the Language Gap: Knowledge Injected Multilingual Question Answeringcited by 5
  25. 2024Resolving Regular Polysemy in Named Entitiescited by 5
  26. 2021Focusing Knowledge-based Graph Argument Mining via Topic Modelingcited by 3
  27. 2021Towards Multi-Sense Cross-Lingual Alignment of Contextual Embeddingscited by 3
  28. 2021WiC = TSV = WSD: On the Equivalence of Three Semantic Taskscited by 3
  29. 2021A Survey of Knowledge Enhanced Pre-trained Modelscited by 3
  30. 2022Does Entity Abstraction Help Generative Transformers Reason?cited by 3
  31. 2022A Survey on Knowledge-Enhanced Pre-trained Language Modelscited by 3
  32. 2023Representation Learning via Variational Bayesian Networkscited by 3
  33. 2023KTRL+F: Knowledge-Augmented In-Document Searchcited by 3
  34. 2025Advances in Pre-trained Language Models for Domain-Specific Text Classification: A Systematic Reviewcited by 3
  35. 2026Beyond Tokens: Concept-Level Training Objectives for LLMscited by 3
  36. 2021BERT Has Uncommon Sense: Similarity Ranking for Word Sense BERTologycited by 2
  37. 2021K-AID: Enhancing Pre-trained Language Models with Domain Knowledge for Question Answeringcited by 2
  38. 2022Improving Contextual Representation with Gloss Regularized Pre-trainingcited by 2

50 most cited of 73 All 73 (JSON)