# Language Models for Lexical Inference in Context

arXiv: https://arxiv.org/abs/2102.05331 · HTML: https://mlsanity.com/paper/2102.05331/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 55 references that are in the map.
- First posted on arXiv 10 Feb 2021.

## Builds on (most cited first, 21 of 21)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference](https://mlsanity.com/paper/1704.05426/) (2017)
- [Language Models as Knowledge Bases?](https://mlsanity.com/paper/1909.01066/) (2019)
- [Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inference](https://mlsanity.com/paper/2001.07676/) (2020)
- [Negated and Misprimed Probes for Pretrained Language Models: Birds Can Talk, But Cannot Fly](https://mlsanity.com/paper/1911.03343/) (2019)
- [Show Your Work: Improved Reporting of Experimental Results](https://mlsanity.com/paper/1909.03004/) (2019)
- [Breaking NLI Systems with Sentences that Require Simple Lexical Inferences](https://mlsanity.com/paper/1805.02266/) (2018)
- [Inducing Relational Knowledge from BERT](https://mlsanity.com/paper/1911.12753/) (2019)
- [Do Neural Language Representations Learn Physical Commonsense?](https://mlsanity.com/paper/1908.02899/) (2019)
- [Hearst Patterns Revisited: Automatic Hypernym Detection from Large Text Corpora](https://mlsanity.com/paper/1806.03191/) (2018)
- [Specialising Word Vectors for Lexical Entailment](https://mlsanity.com/paper/1710.06371/) (2017)
- [Word Sense Induction with Neural biLM and Symmetric Patterns](https://mlsanity.com/paper/1808.08518/) (2018)
- [Probabilistic Models of Relational Implication](https://mlsanity.com/paper/1907.12048/) (2019)
- [Paraphrasing vs Coreferring: Two Sides of the Same Coin](https://mlsanity.com/paper/2004.14979/) (2020)
- [SherLIiC: A Typed Event-Focused Lexical Inference Benchmark for Evaluating Natural Language Inference](https://mlsanity.com/paper/1906.01393/) (2019)
- [Relations such as Hypernymy: Identifying and Exploiting Hearst Patterns in Distributional Vectors for Lexical Entailment](https://mlsanity.com/paper/1605.05433/) (2016)

## Built on by (most cited first, 9 of 9)

- [Sources of Hallucination by Large Language Models on Inference Tasks](https://mlsanity.com/paper/2305.14552/) (2023)
- [Entailment Graph Learning with Textual Entailment and Soft Transitivity](https://mlsanity.com/paper/2204.03286/) (2022)
- [Smoothing Entailment Graphs with Language Models](https://mlsanity.com/paper/2208.00318/) (2022)
- [Cross-lingual Inference with A Chinese Entailment Graph](https://mlsanity.com/paper/2203.06264/) (2022)
- [Language Models Are Poor Learners of Directional Inference](https://mlsanity.com/paper/2210.04695/) (2022)
- [Continuous Entailment Patterns for Lexical Inference in Context](https://mlsanity.com/paper/2109.03695/) (2021)
- [Curriculum: A Broad-Coverage Benchmark for Linguistic Phenomena in Natural Language Understanding](https://mlsanity.com/paper/2204.06283/) (2022)
- [S2LPP: Small-to-Large Prompt Prediction across LLMs](https://mlsanity.com/paper/2505.20097/) (2025)
- [LLMs are Frequency Pattern Learners in Natural Language Inference](https://mlsanity.com/paper/2505.21011/) (2025)

## Abstract (arXiv)

Lexical inference in context (LIiC) is the task of recognizing textual entailment between two very similar sentences, i.e., sentences that only differ in one expression. It can therefore be seen as a variant of the natural language inference task that is focused on lexical semantics. We formulate and evaluate the first approaches based on pretrained language models (LMs) for this task: (i) a few-shot NLI classifier, (ii) a relation induction approach based on handcrafted patterns expressing the semantics of lexical inference, and (iii) a variant of (ii) with patterns that were automatically extracted from a corpus. All our approaches outperform the previous state of the art, showing the potential of pretrained LMs for LIiC. In an extensive analysis, we investigate factors of success and failure of our three approaches.