# Card-660: Cambridge Rare Word Dataset - a Reliable Benchmark for Infrequent Word Representation Models

arXiv: https://arxiv.org/abs/1808.09308 · HTML: https://mlsanity.com/paper/1808.09308/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 Aug 2018.

## Built on by (most cited first, 11 of 11)

- [Fast, Effective, and Self-Supervised: Transforming Masked Language Models into Universal Lexical and Sentence Encoders](https://mlsanity.com/paper/2104.08027/) (2021)
- [Can a Fruit Fly Learn Word Embeddings?](https://mlsanity.com/paper/2101.06887/) (2021)
- [The Impact of Word Splitting on the Semantic Content of Contextualized Word Representations](https://mlsanity.com/paper/2402.14616/) (2024)
- [IMAGINATOR: Pre-Trained Image+Text Joint Embeddings using Word-Level Grounding of Images](https://mlsanity.com/paper/2305.10438/) (2023)
- [RetroGAN: A Cyclic Post-Specialization System for Improving Out-of-Knowledge and Rare Word Representations](https://mlsanity.com/paper/2108.12941/) (2021)
- [Multi-Sense Embeddings for Language Models and Knowledge Distillation](https://mlsanity.com/paper/2504.06036/) (2025)
- [Evaluation of taxonomic and neural embedding methods for calculating semantic similarity](https://mlsanity.com/paper/2209.15197/) (2022)
- [Lexical semantics enhanced neural word embeddings](https://mlsanity.com/paper/2210.00754/) (2022)
- [MultiHashFormer: Hash-based Generative Language Models](https://mlsanity.com/paper/2606.28057/) (2026)
- [RESTORE: Graph Embedding Assessment Through Reconstruction](https://mlsanity.com/paper/2308.14659/) (2023)
- [Predicting Word Similarity in Context with Referential Translation Machines](https://mlsanity.com/paper/2407.06230/) (2024)

## Abstract (arXiv)

Rare word representation has recently enjoyed a surge of interest, owing to the crucial role that effective handling of infrequent words can play in accurate semantic understanding. However, there is a paucity of reliable benchmarks for evaluation and comparison of these techniques. We show in this paper that the only existing benchmark (the Stanford Rare Word dataset) suffers from low-confidence annotations and limited vocabulary; hence, it does not constitute a solid comparison framework. In order to fill this evaluation gap, we propose CAmbridge Rare word Dataset (Card-660), an expert-annotated word similarity dataset which provides a highly reliable, yet challenging, benchmark for rare word representation techniques. Through a set of experiments we show that even the best mainstream word embeddings, with millions of words in their vocabularies, are unable to achieve performances higher than 0.43 (Pearson correlation) on the dataset, compared to a human-level upperbound of 0.90. We release the dataset and the annotation materials at https://pilehvar.github.io/card-660/.