# SP-10K: A Large-scale Evaluation Set for Selectional Preference Acquisition

arXiv: https://arxiv.org/abs/1906.02123 · HTML: https://mlsanity.com/paper/1906.02123/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 May 2019.

## Built on by (most cited first, 8 of 8)

- [Back to Square One: Artifact Detection, Training and Commonsense Disentanglement in the Winograd Schema](https://mlsanity.com/paper/2104.08161/) (2021)
- [ASER: Towards Large-scale Commonsense Knowledge Acquisition via Higher-order Selectional Preference over Eventualities](https://mlsanity.com/paper/2104.02137/) (2021)
- [Modeling Event Plausibility with Consistent Conceptual Abstraction](https://mlsanity.com/paper/2104.10247/) (2021)
- [Did the Cat Drink the Coffee? Challenging Transformers with Generalized Event Knowledge](https://mlsanity.com/paper/2107.10922/) (2021)
- [A Dataset for Physical and Abstract Plausibility and Sources of Human Disagreement](https://mlsanity.com/paper/2404.04035/) (2024)
- [Structural Ambiguity and its Disambiguation in Language Model Based Parsers: the Case of Dutch Clause Relativization](https://mlsanity.com/paper/2305.14917/) (2023)
- [Solving the Challenge Set without Solving the Task: On Winograd Schemas as a Test of Pronominal Coreference Resolution](https://mlsanity.com/paper/2410.09448/) (2024)
- [Cross-Document Contextual Coreference Resolution in Knowledge Graphs](https://mlsanity.com/paper/2504.05767/) (2025)

## Abstract (arXiv)

Selectional Preference (SP) is a commonly observed language phenomenon and proved to be useful in many natural language processing tasks. To provide a better evaluation method for SP models, we introduce SP-10K, a large-scale evaluation set that provides human ratings for the plausibility of 10,000 SP pairs over five SP relations, covering 2,500 most frequent verbs, nouns, and adjectives in American English. Three representative SP acquisition methods based on pseudo-disambiguation are evaluated with SP-10K. To demonstrate the importance of our dataset, we investigate the relationship between SP-10K and the commonsense knowledge in ConceptNet5 and show the potential of using SP to represent the commonsense knowledge. We also use the Winograd Schema Challenge to prove that the proposed new SP relations are essential for the hard pronoun coreference resolution problem.