# Always Keep your Target in Mind: Studying Semantics and Improving Performance of Neural Lexical Substitution

arXiv: https://arxiv.org/abs/2206.11815 · HTML: https://mlsanity.com/paper/2206.11815/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 27 references that are in the map.
- First posted on arXiv 7 Jun 2022.

## Builds on (most cited first, 8 of 8)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://mlsanity.com/paper/1906.08237/) (2019)
- [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018)
- [Rare Words: A Major Problem for Contextualized Embeddings And How to Fix it by Attentive Mimicking](https://mlsanity.com/paper/1904.06707/) (2019)
- [Word Sense Induction with Neural biLM and Symmetric Patterns](https://mlsanity.com/paper/1808.08518/) (2018)
- [Towards better substitution-based word sense induction](https://mlsanity.com/paper/1905.12598/) (2019)

## Built on by (most cited first, 12 of 12)

- [Language Model Behavior: A Comprehensive Survey](https://mlsanity.com/paper/2303.11504/) (2023)
- [Language Models are Surprisingly Fragile to Drug Names in Biomedical Benchmarks](https://mlsanity.com/paper/2406.12066/) (2024)
- [Unsupervised Lexical Substitution with Decontextualised Embeddings](https://mlsanity.com/paper/2209.08236/) (2022)
- [Deep Learning Approaches to Lexical Simplification: A Survey](https://mlsanity.com/paper/2305.12000/) (2023)
- [TDG: Text-guided Domain Generalization](https://mlsanity.com/paper/2308.09931/) (2023)
- [Multilingual Substitution-based Word Sense Induction](https://mlsanity.com/paper/2405.11086/) (2024)
- [Implicit Causality-biases in humans and LLMs as a tool for benchmarking LLM discourse capabilities](https://mlsanity.com/paper/2501.12980/) (2025)
- [Learning Concepts, Not Tokens: Self-Supervised Semantic Alignment for Language Models](https://mlsanity.com/paper/2603.29123/) (2026)
- [Automatic Text Simplification of News Articles in the Context of Public Broadcasting](https://mlsanity.com/paper/2212.13317/) (2022)
- [Deep-change at AXOLOTL-24: Orchestrating WSD and WSI Models for Semantic Change Modeling](https://mlsanity.com/paper/2408.05184/) (2024)
- [Unsupervised Candidate Ranking for Lexical Substitution via Holistic Sentence Semantics](https://mlsanity.com/paper/2509.11513/) (2025)
- [Precision in Practice: Knowledge Guided Code Summarizing Grounded in Industrial Expectations](https://mlsanity.com/paper/2602.03400/) (2026)

## Abstract (arXiv)

Lexical substitution, i.e. generation of plausible words that can replace a particular target word in a given context, is an extremely powerful technology that can be used as a backbone of various NLP applications, including word sense induction and disambiguation, lexical relation extraction, data augmentation, etc. In this paper, we present a large-scale comparative study of lexical substitution methods employing both rather old and most recent language and masked language models (LMs and MLMs), such as context2vec, ELMo, BERT, RoBERTa, XLNet. We show that already competitive results achieved by SOTA LMs/MLMs can be further substantially improved if information about the target word is injected properly. Several existing and new target word injection methods are compared for each LM/MLM using both intrinsic evaluation on lexical substitution datasets and extrinsic evaluation on word sense induction (WSI) datasets. On two WSI datasets we obtain new SOTA results. Besides, we analyze the types of semantic relations between target words and their substitutes generated by different models or given by annotators.