# Unsupervised Candidate Ranking for Lexical Substitution via Holistic Sentence Semantics

arXiv: https://arxiv.org/abs/2509.11513 · HTML: https://mlsanity.com/paper/2509.11513/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 12 of its 17 references that are in the map.
- First posted on arXiv 15 Sep 2025.

## Builds on (most cited first, 12 of 12)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Axiomatic Attribution for Deep Networks](https://mlsanity.com/paper/1703.01365/) (2017)
- [SimCSE: Simple Contrastive Learning of Sentence Embeddings](https://mlsanity.com/paper/2104.08821/) (2021)
- [DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharing](https://mlsanity.com/paper/2111.09543/) (2021)
- [MPNet: Masked and Permuted Pre-training for Language Understanding](https://mlsanity.com/paper/2004.09297/) (2020)
- [Why Attentions May Not Be Interpretable?](https://mlsanity.com/paper/2006.05656/) (2020)
- [Swords: A Benchmark for Lexical Substitution with Improved Data Coverage and Quality](https://mlsanity.com/paper/2106.04102/) (2021)
- [Always Keep your Target in Mind: Studying Semantics and Improving Performance of Neural Lexical Substitution](https://mlsanity.com/paper/2206.11815/) (2022)
- [Unsupervised Lexical Substitution with Decontextualised Embeddings](https://mlsanity.com/paper/2209.08236/) (2022)
- [LexSubCon: Integrating Knowledge from Lexical Resources into Contextual Embeddings for Lexical Substitution](https://mlsanity.com/paper/2107.05132/) (2021)
- [ParaLS: Lexical Substitution via Pretrained Paraphraser](https://mlsanity.com/paper/2305.08146/) (2023)
- [Self-Supervised Contrastive Learning with Adversarial Perturbations for Defending Word Substitution-based Attacks](https://mlsanity.com/paper/2107.07610/) (2021)

## Abstract (arXiv)

A key subtask in lexical substitution is ranking the given candidate words. A common approach is to replace the target word with a candidate in the original sentence and feed the modified sentence into a model to capture semantic differences before and after substitution. However, effectively modeling the bidirectional influence of candidate substitution on both the target word and its context remains challenging. Existing methods often focus solely on semantic changes at the target position or rely on parameter tuning over multiple evaluation metrics, making it difficult to accurately characterize semantic variation. To address this, we investigate two approaches: one based on attention weights and another leveraging the more interpretable integrated gradients method, both designed to measure the influence of context tokens on the target token and to rank candidates by incorporating semantic similarity between the original and substituted sentences. Experiments on the LS07 and SWORDS datasets demonstrate that both approaches improve ranking performance.