# LexSubCon: Integrating Knowledge from Lexical Resources into Contextual Embeddings for Lexical Substitution

arXiv: https://arxiv.org/abs/2107.05132 · HTML: https://mlsanity.com/paper/2107.05132/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 47 references that are in the map.
- First posted on arXiv 11 Jul 2021.

## Builds on (most cited first, 16 of 16)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://mlsanity.com/paper/1906.08237/) (2019)
- [ALBERT: A Lite BERT for Self-supervised Learning of Language Representations](https://mlsanity.com/paper/1909.11942/) (2019)
- [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018)
- [HuggingFace's Transformers: State-of-the-art Natural Language Processing](https://mlsanity.com/paper/1910.03771/) (2019)
- [EDA: Easy Data Augmentation Techniques for Boosting Performance on Text Classification Tasks](https://mlsanity.com/paper/1901.11196/) (2019)
- [Improving Neural Machine Translation Models with Monolingual Data](https://mlsanity.com/paper/1511.06709/) (2015)
- [Show Your Work: Improved Reporting of Experimental Results](https://mlsanity.com/paper/1909.03004/) (2019)
- [GlossBERT: BERT for Word Sense Disambiguation with Gloss Knowledge](https://mlsanity.com/paper/1908.07245/) (2019)
- [SenseBERT: Driving Some Sense into BERT](https://mlsanity.com/paper/1908.05646/) (2019)
- [Language Modelling Makes Sense: Propagating Representations through WordNet for Full-Coverage Word Sense Disambiguation](https://mlsanity.com/paper/1906.10007/) (2019)
- [Learning to Compute Word Embeddings On the Fly](https://mlsanity.com/paper/1706.00286/) (2017)
- [A Comparative Study of Lexical Substitution Approaches based on Neural Language Models](https://mlsanity.com/paper/2006.00031/) (2020)

## Built on by (most cited first, 6 of 6)

- [Learning to Substitute Words with Model-based Score Ranking](https://mlsanity.com/paper/2502.05933/) (2025)
- [Unsupervised Lexical Substitution with Decontextualised Embeddings](https://mlsanity.com/paper/2209.08236/) (2022)
- [ParaLS: Lexical Substitution via Pretrained Paraphraser](https://mlsanity.com/paper/2305.08146/) (2023)
- [Smart Word Suggestions for Writing Assistance](https://mlsanity.com/paper/2305.09975/) (2023)
- [Lexical Substitution is not Synonym Substitution: On the Importance of Producing Contextually Relevant Word Substitutes](https://mlsanity.com/paper/2502.04173/) (2025)
- [Unsupervised Candidate Ranking for Lexical Substitution via Holistic Sentence Semantics](https://mlsanity.com/paper/2509.11513/) (2025)

## Abstract (arXiv)

Lexical substitution is the task of generating meaningful substitutes for a word in a given textual context. Contextual word embedding models have achieved state-of-the-art results in the lexical substitution task by relying on contextual information extracted from the replaced word within the sentence. However, such models do not take into account structured knowledge that exists in external lexical databases. We introduce LexSubCon, an end-to-end lexical substitution framework based on contextual embedding models that can identify highly accurate substitute candidates. This is achieved by combining contextual information with knowledge from structured lexical resources. Our approach involves: (i) introducing a novel mix-up embedding strategy in the creation of the input embedding of the target word through linearly interpolating the pair of the target input embedding and the average embedding of its probable synonyms; (ii) considering the similarity of the sentence-definition embeddings of the target word and its proposed candidates; and, (iii) calculating the effect of each substitution in the semantics of the sentence through a fine-tuned sentence similarity model. Our experiments show that LexSubCon outperforms previous state-of-the-art methods on LS07 and CoInCo benchmark datasets that are widely used for lexical substitution tasks.