# HyperLex: A Large-Scale Evaluation of Graded Lexical Entailment

arXiv: https://arxiv.org/abs/1608.02117 · HTML: https://mlsanity.com/paper/1608.02117/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 6 Aug 2016.

## Built on by (most cited first, 26 of 26)

- [Hyperbolic Deep Neural Networks: A Survey](https://mlsanity.com/paper/2101.04562/) (2021)
- [Meaning without reference in large language models](https://mlsanity.com/paper/2208.02957/) (2022)
- [Emergent Visual-Semantic Hierarchies in Image-Text Representations](https://mlsanity.com/paper/2407.08521/) (2024)
- [From superposition to sparse codes: interpretable representations in neural networks](https://mlsanity.com/paper/2503.01824/) (2025)
- [TaxoLLaMA: WordNet-based Model for Solving Multiple Lexical Semantic Tasks](https://mlsanity.com/paper/2403.09207/) (2024)
- [ALL Dolphins Are Intelligent and SOME Are Friendly: Probing BERT for Nouns' Semantic Properties and their Prototypicality](https://mlsanity.com/paper/2110.06376/) (2021)
- [Competence-Based Analysis of Language Models](https://mlsanity.com/paper/2303.00333/) (2023)
- [Hierarchical Retrieval: The Geometry and a Pretrain-Finetune Recipe](https://mlsanity.com/paper/2509.16411/) (2025)
- [AfriVEC: Word Embedding Models for African Languages. Case Study of Fon and Nobiin](https://mlsanity.com/paper/2103.05132/) (2021)
- [Cross-lingual Word Embeddings in Hyperbolic Space](https://mlsanity.com/paper/2205.01907/) (2022)
- [On the Distinctive Co-occurrence Characteristics of Antonymy](https://mlsanity.com/paper/2509.11534/) (2025)
- [HiMo-CLIP: Modeling Semantic Hierarchy and Monotonicity in Vision-Language Alignment](https://mlsanity.com/paper/2511.06653/) (2025)
- [Lexical semantics enhanced neural word embeddings](https://mlsanity.com/paper/2210.00754/) (2022)
- [Probing Taxonomic and Thematic Embeddings for Taxonomic Information](https://mlsanity.com/paper/2301.10656/) (2023)
- [Exploring Category Structure with Contextual Language Models and Lexical Semantic Networks](https://mlsanity.com/paper/2302.06942/) (2023)
- [A RelEntLess Benchmark for Modelling Graded Relations between Named Entities](https://mlsanity.com/paper/2305.15002/) (2023)
- [3D-EX : A Unified Dataset of Definitions and Dictionary Examples](https://mlsanity.com/paper/2308.03043/) (2023)
- [Talking the Talk Does Not Entail Walking the Walk: On the Limits of Large Language Models in Lexical Entailment Recognition](https://mlsanity.com/paper/2406.14894/) (2024)
- [Misalignment of Semantic Relation Knowledge between WordNet and Human Intuition](https://mlsanity.com/paper/2412.02138/) (2024)
- [One Word Is Not Enough: Simple Prompts Improve Word Embeddings](https://mlsanity.com/paper/2512.06744/) (2025)
- [QuantumPhaseNet: A Gauge-Covariant Geometric and Quantum-Spectral Theory of Semantic Concept Hierarchies with Prototype Validation of a Classical Quantum-Inspired Model](https://mlsanity.com/paper/2608.15820/) (2026)
- [Inferring Adjective Hypernyms with Language Models to Increase the Connectivity of Open English Wordnet](https://mlsanity.com/paper/2506.10715/) (2025)
- [Accurate and Efficient Statistical Testing for Word Semantic Breadth](https://mlsanity.com/paper/2605.08048/) (2026)
- [Role-Aware Neural Convex Divergence Heads for Asymmetric Representation Learning](https://mlsanity.com/paper/2607.01762/) (2026)
- [Explaining Textual Entailment with Lexical Entailments: Using LLMs to Supply Lexical Relations for Formal Proofs](https://mlsanity.com/paper/2609.32491/) (2026)
- [HodgePhaseNet: Discrete Hodge Decomposition of Semantic Flows in Transformer Attention](https://mlsanity.com/paper/2610.14971/) (2026)

## Abstract (arXiv)

We introduce HyperLex - a dataset and evaluation resource that quantifies the extent of of the semantic category membership, that is, type-of relation also known as hyponymy-hypernymy or lexical entailment (LE) relation between 2,616 concept pairs. Cognitive psychology research has established that typicality and category/class membership are computed in human semantic memory as a gradual rather than binary relation. Nevertheless, most NLP research, and existing large-scale invetories of concept category membership (WordNet, DBPedia, etc.) treat category membership and LE as binary. To address this, we asked hundreds of native English speakers to indicate typicality and strength of category membership between a diverse range of concept pairs on a crowdsourcing platform. Our results confirm that category membership and LE are indeed more gradual than binary. We then compare these human judgements with the predictions of automatic systems, which reveals a huge gap between human performance and state-of-the-art LE, distributional and representation learning models, and substantial differences between the models themselves. We discuss a pathway for improving semantic models to overcome this discrepancy, and indicate future application areas for improved graded LE systems.