# Talking the Talk Does Not Entail Walking the Walk: On the Limits of Large Language Models in Lexical Entailment Recognition

arXiv: https://arxiv.org/abs/2406.14894 · HTML: https://mlsanity.com/paper/2406.14894/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 35 references that are in the map.
- First posted on arXiv 21 Jun 2024.

## Builds on (most cited first, 17 of 17)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling](https://mlsanity.com/paper/2304.01373/) (2023)
- [The Falcon Series of Open Language Models](https://mlsanity.com/paper/2311.16867/) (2023)
- [Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Survey](https://mlsanity.com/paper/2111.01243/) (2021)
- [Language Modelling Makes Sense: Propagating Representations through WordNet for Full-Coverage Word Sense Disambiguation](https://mlsanity.com/paper/1906.10007/) (2019)
- [Say What You Mean! Large Language Models Speak Too Positively about Negative Commonsense Knowledge](https://mlsanity.com/paper/2305.05976/) (2023)
- [Prompting Frameworks for Large Language Models: A Survey](https://mlsanity.com/paper/2311.12785/) (2023)
- [HyperLex: A Large-Scale Evaluation of Graded Lexical Entailment](https://mlsanity.com/paper/1608.02117/) (2016)
- [TaxoLLaMA: WordNet-based Model for Solving Multiple Lexical Semantic Tasks](https://mlsanity.com/paper/2403.09207/) (2024)
- [What do Language Models know about word senses? Zero-Shot WSD with Language Models and Domain Inventories](https://mlsanity.com/paper/2302.03353/) (2023)
- [Better Language Model with Hypernym Class Prediction](https://mlsanity.com/paper/2203.10692/) (2022)
- [Probing Pretrained Language Models with Hierarchy Properties](https://mlsanity.com/paper/2312.09670/) (2023)
- [Vec2Gloss: definition modeling leveraging contextualized vectors with Wordnet gloss](https://mlsanity.com/paper/2305.17855/) (2023)

## Built on by (most cited first, 1 of 1)

- [Investigating Neurons and Heads in Transformer-based LLMs for Typographical Errors](https://mlsanity.com/paper/2502.19669/) (2025)

## Abstract (arXiv)

Verbs form the backbone of language, providing the structure and meaning to sentences. Yet, their intricate semantic nuances pose a longstanding challenge. Understanding verb relations through the concept of lexical entailment is crucial for comprehending sentence meanings and grasping verb dynamics. This work investigates the capabilities of eight Large Language Models in recognizing lexical entailment relations among verbs through differently devised prompting strategies and zero-/few-shot settings over verb pairs from two lexical databases, namely WordNet and HyperLex. Our findings unveil that the models can tackle the lexical entailment recognition task with moderately good performance, although at varying degree of effectiveness and under different conditions. Also, utilizing few-shot prompting can enhance the models' performance. However, perfectly solving the task arises as an unmet challenge for all examined LLMs, which raises an emergence for further research developments on this topic.