# Swords: A Benchmark for Lexical Substitution with Improved Data Coverage and Quality

arXiv: https://arxiv.org/abs/2106.04102 · HTML: https://mlsanity.com/paper/2106.04102/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 7 of its 43 references that are in the map.
- First posted on arXiv 8 Jun 2021.

## Builds on (most cited first, 7 of 7)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Adversarial Example Generation with Syntactically Controlled Paraphrase Networks](https://mlsanity.com/paper/1804.06059/) (2018)
- [What Can We Learn from Collective Human Opinions on Natural Language Inference Data?](https://mlsanity.com/paper/2010.03532/) (2020)
- [ParaNMT-50M: Pushing the Limits of Paraphrastic Sentence Embeddings with Millions of Machine Translations](https://mlsanity.com/paper/1711.05732/) (2017)
- [ParaBank: Monolingual Bitext Generation and Sentential Paraphrasing via Lexically-constrained Neural Machine Translation](https://mlsanity.com/paper/1901.03644/) (2019)
- [A Comparative Study of Lexical Substitution Approaches based on Neural Language Models](https://mlsanity.com/paper/2006.00031/) (2020)

## Built on by (most cited first, 13 of 13)

- [Tracing Text Provenance via Context-Aware Lexical Substitution](https://mlsanity.com/paper/2112.07873/) (2021)
- [P-Adapters: Robustly Extracting Factual Information from Language Models with Diverse Prompts](https://mlsanity.com/paper/2110.07280/) (2021)
- [Probing Semantic Routing in Large Mixture-of-Expert Models](https://mlsanity.com/paper/2502.10928/) (2025)
- [Unsupervised Lexical Substitution with Decontextualised Embeddings](https://mlsanity.com/paper/2209.08236/) (2022)
- [ParaLS: Lexical Substitution via Pretrained Paraphraser](https://mlsanity.com/paper/2305.08146/) (2023)
- [Smart Word Suggestions for Writing Assistance](https://mlsanity.com/paper/2305.09975/) (2023)
- [RankAlign: A Ranking View of the Generator-Validator Gap in Large Language Models](https://mlsanity.com/paper/2504.11381/) (2025)
- [ProLex: A Benchmark for Language Proficiency-oriented Lexical Substitution](https://mlsanity.com/paper/2401.11356/) (2024)
- [ChiKhaPo: A Large-Scale Multilingual Benchmark for Evaluating Lexical Comprehension and Generation in Large Language Models](https://mlsanity.com/paper/2510.16928/) (2025)
- [Unsupervised Paraphrasing of Multiword Expressions](https://mlsanity.com/paper/2306.01443/) (2023)
- [Unsupervised Lexical Simplification with Context Augmentation](https://mlsanity.com/paper/2311.00310/) (2023)
- [Lexical Substitution is not Synonym Substitution: On the Importance of Producing Contextually Relevant Word Substitutes](https://mlsanity.com/paper/2502.04173/) (2025)
- [Unsupervised Candidate Ranking for Lexical Substitution via Holistic Sentence Semantics](https://mlsanity.com/paper/2509.11513/) (2025)

## Abstract (arXiv)

We release a new benchmark for lexical substitution, the task of finding appropriate substitutes for a target word in a context. To assist humans with writing, lexical substitution systems can suggest words that humans cannot easily think of. However, existing benchmarks depend on human recall as the only source of data, and therefore lack coverage of the substitutes that would be most helpful to humans. Furthermore, annotators often provide substitutes of low quality, which are not actually appropriate in the given context. We collect higher-coverage and higher-quality data by framing lexical substitution as a classification problem, guided by the intuition that it is easier for humans to judge the appropriateness of candidate substitutes than conjure them from memory. To this end, we use a context-free thesaurus to produce candidates and rely on human judgement to determine contextual appropriateness. Compared to the previous largest benchmark, our Swords benchmark has 4.1x more substitutes per target word for the same level of quality, and its substitutes are 1.5x more appropriate (based on human judgement) for the same number of substitutes.