# SimVerb-3500: A Large-Scale Evaluation Set of Verb Similarity

arXiv: https://arxiv.org/abs/1608.00869 · HTML: https://mlsanity.com/paper/1608.00869/

## Key facts

- Cited by 44 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Aug 2016.

## Built on by (most cited first, 30 of 44)

- [All Bark and No Bite: Rogue Dimensions in Transformer Language Models Obscure Representational Quality](https://mlsanity.com/paper/2109.04404/) (2021)
- [The Zero Resource Speech Challenge 2021: Spoken language modelling](https://mlsanity.com/paper/2104.14700/) (2021)
- [Self-supervised language learning from raw audio: Lessons from the Zero Resource Speech Challenge](https://mlsanity.com/paper/2210.15759/) (2022)
- [Is Child-Directed Speech Effective Training Data for Language Models?](https://mlsanity.com/paper/2408.03617/) (2024)
- [Learning to Remove: Towards Isotropic Pre-trained BERT Embedding](https://mlsanity.com/paper/2104.05274/) (2021)
- [Just Rank: Rethinking Evaluation with Word and Sentence Similarities](https://mlsanity.com/paper/2203.02679/) (2022)
- [Discovering Universal Geometry in Embeddings with ICA](https://mlsanity.com/paper/2305.13175/) (2023)
- [Obtaining Better Static Word Embeddings Using Contextual Embedding Models](https://mlsanity.com/paper/2106.04302/) (2021)
- [Visual Grounding Helps Learn Word Meanings in Low-Data Regimes](https://mlsanity.com/paper/2310.13257/) (2023)
- [Learning Fair Representations via Rate-Distortion Maximization](https://mlsanity.com/paper/2202.00035/) (2022)
- [Backpack Language Models](https://mlsanity.com/paper/2305.16765/) (2023)
- [Contrastive Visual Semantic Pretraining Magnifies the Semantics of Natural Language Representations](https://mlsanity.com/paper/2203.07511/) (2022)
- [Word Embeddings via Causal Inference: Gender Bias Reducing and Semantic Information Preserving](https://mlsanity.com/paper/2112.05194/) (2021)
- [RAW-C: Relatedness of Ambiguous Words--in Context (A New Lexical Resource for English)](https://mlsanity.com/paper/2105.13266/) (2021)
- [Learning Zero-Shot Multifaceted Visually Grounded Word Embeddings via Multi-Task Training](https://mlsanity.com/paper/2104.07500/) (2021)
- [Are word boundaries useful for unsupervised language learning?](https://mlsanity.com/paper/2210.02956/) (2022)
- [Multi-sense embeddings through a word sense disambiguation process](https://mlsanity.com/paper/2101.08700/) (2021)
- [On the Interplay Between Fine-tuning and Composition in Transformers](https://mlsanity.com/paper/2105.14668/) (2021)
- [Lexicon-Level Contrastive Visual-Grounding Improves Language Modeling](https://mlsanity.com/paper/2403.14551/) (2024)
- [Language with Vision: a Study on Grounded Word and Sentence Embeddings](https://mlsanity.com/paper/2206.08823/) (2022)
- [Bringing Up a Bilingual BabyLM: Investigating Multilingual Language Acquisition Using Small-Scale Models](https://mlsanity.com/paper/2603.29552/) (2026)
- [Lacking the embedding of a word? Look it up into a traditional dictionary](https://mlsanity.com/paper/2109.11763/) (2021)
- [Multi-Relational Hyperbolic Word Embeddings from Natural Language Definitions](https://mlsanity.com/paper/2305.07303/) (2023)
- [A Neighbourhood-Aware Differential Privacy Mechanism for Static Word Embeddings](https://mlsanity.com/paper/2309.10551/) (2023)
- [A Distributional Perspective on Word Learning in Neural Language Models](https://mlsanity.com/paper/2502.05892/) (2025)
- [RetroGAN: A Cyclic Post-Specialization System for Improving Out-of-Knowledge and Rare Word Representations](https://mlsanity.com/paper/2108.12941/) (2021)
- [Are ELECTRA's Sentence Embeddings Beyond Repair? The Case of Semantic Textual Similarity](https://mlsanity.com/paper/2402.13130/) (2024)
- [Measuring Contextual Informativeness in Child-Directed Text](https://mlsanity.com/paper/2412.17427/) (2024)
- [Multi-Sense Embeddings for Language Models and Knowledge Distillation](https://mlsanity.com/paper/2504.06036/) (2025)
- [Identifying and Mitigating Gender Bias in Hyperbolic Word Embeddings](https://mlsanity.com/paper/2109.13767/) (2021)

## Abstract (arXiv)

Verbs play a critical role in the meaning of sentences, but these ubiquitous words have received little attention in recent distributional semantics research. We introduce SimVerb-3500, an evaluation resource that provides human ratings for the similarity of 3,500 verb pairs. SimVerb-3500 covers all normed verb types from the USF free-association database, providing at least three examples for every VerbNet class. This broad coverage facilitates detailed analyses of how syntactic and semantic phenomena together influence human understanding of verb meaning. Further, with significantly larger development and test sets than existing benchmarks, SimVerb-3500 enables more robust evaluation of representation learning architectures and promotes the development of methods tailored to verbs. We hope that SimVerb-3500 will enable a richer understanding of the diversity and complexity of verb semantics and guide the development of systems that can effectively represent and interpret this meaning.