# PolyBERT: Fine-Tuned Poly Encoder BERT-Based Model for Word Sense Disambiguation

arXiv: https://arxiv.org/abs/2506.00968 · HTML: https://mlsanity.com/paper/2506.00968/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 6 of its 18 references that are in the map.
- First posted on arXiv 1 Jun 2025.

## Builds on (most cited first, 6 of 6)

- [GlossBERT: BERT for Word Sense Disambiguation with Gloss Knowledge](https://mlsanity.com/paper/1908.07245/) (2019)
- [Moving Down the Long Tail of Word Sense Disambiguation with Gloss-Informed Biencoders](https://mlsanity.com/paper/2005.02590/) (2020)
- [Improved Word Sense Disambiguation Using Pre-Trained Contextualized Word Representations](https://mlsanity.com/paper/1910.00194/) (2019)
- [Sense Vocabulary Compression through the Semantic Knowledge of WordNet for Neural Word Sense Disambiguation](https://mlsanity.com/paper/1905.05677/) (2019)
- [Adapting BERT for Word Sense Disambiguation with Gloss Selection Objective and Example Sentences](https://mlsanity.com/paper/2009.11795/) (2020)
- [Word Sense Disambiguation using a Bidirectional LSTM](https://mlsanity.com/paper/1606.03568/) (2016)

## Built on by (most cited first, 1 of 1)

- [ViConBERT: Context-Gloss Aligned Vietnamese Word Embedding for Polysemous and Sense-Aware Representations](https://mlsanity.com/paper/2511.12249/) (2025)

## Abstract (arXiv)

Mainstream Word Sense Disambiguation (WSD) approaches have employed BERT to extract semantics from both context and definitions of senses to determine the most suitable sense of a target word, achieving notable performance. However, there are two limitations in these approaches. First, previous studies failed to balance the representation of token-level (local) and sequence-level (global) semantics during feature extraction, leading to insufficient semantic representation and a performance bottleneck. Second, these approaches incorporated all possible senses of each target word during the training phase, leading to unnecessary computational costs. To overcome these limitations, this paper introduces a poly-encoder BERT-based model with batch contrastive learning for WSD, named PolyBERT. Compared with previous WSD methods, PolyBERT has two improvements: (1) A poly-encoder with a multi-head attention mechanism is utilized to fuse token-level (local) and sequence-level (global) semantics, rather than focusing on just one. This approach enriches semantic representation by balancing local and global semantics. (2) To avoid redundant training inputs, Batch Contrastive Learning (BCL) is introduced. BCL utilizes the correct senses of other target words in the same batch as negative samples for the current target word, which reduces training inputs and computational cost. The experimental results demonstrate that PolyBERT outperforms baseline WSD methods such as Huang's GlossBERT and Blevins's BEM by 2% in F1-score. In addition, PolyBERT with BCL reduces GPU hours by 37.6% compared with PolyBERT without BCL.