# RAW-C: Relatedness of Ambiguous Words--in Context (A New Lexical Resource for English)

arXiv: https://arxiv.org/abs/2105.13266 · HTML: https://mlsanity.com/paper/2105.13266/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 51 references that are in the map.
- First posted on arXiv 27 May 2021.

## Builds on (most cited first, 15 of 15)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://mlsanity.com/paper/1906.08237/) (2019)
- [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018)
- [WiC: the Word-in-Context Dataset for Evaluating Context-Sensitive Meaning Representations](https://mlsanity.com/paper/1808.09121/) (2018)
- [AllenNLP: A Deep Semantic Natural Language Processing Platform](https://mlsanity.com/paper/1803.07640/) (2018)
- [Experience Grounds Language](https://mlsanity.com/paper/2004.10151/) (2020)
- [SimLex-999: Evaluating Semantic Models with (Genuine) Similarity Estimation](https://mlsanity.com/paper/1408.3456/) (2014)
- [SenseBERT: Driving Some Sense into BERT](https://mlsanity.com/paper/1908.05646/) (2019)
- [Does BERT Make Any Sense? Interpretable Word Sense Disambiguation with Contextualized Embeddings](https://mlsanity.com/paper/1909.10430/) (2019)
- [SimVerb-3500: A Large-Scale Evaluation Set of Verb Similarity](https://mlsanity.com/paper/1608.00869/) (2016)
- [Contextualized Word Embeddings Encode Aspects of Human-Like Word Sense Knowledge](https://mlsanity.com/paper/2010.13057/) (2020)
- [(Re)construing Meaning in NLP](https://mlsanity.com/paper/2005.09099/) (2020)
- [Putting words in context: LSTM language models and lexical ambiguity](https://mlsanity.com/paper/1906.05149/) (2019)
- [Analysis and Evaluation of Language Models for Word Sense Disambiguation](https://mlsanity.com/paper/2008.11608/) (2020)
- [Non-distributional Word Vector Representations](https://mlsanity.com/paper/1506.05230/) (2015)

## Built on by (most cited first, 10 of 10)

- [Do Large Language Models know what humans know?](https://mlsanity.com/paper/2209.01515/) (2022)
- [Putting Words in BERT's Mouth: Navigating Contextualized Vector Spaces with Pseudowords](https://mlsanity.com/paper/2109.11491/) (2021)
- [Evaluating Contextualized Representations of (Spanish) Ambiguous Words: A New Lexical Resource and Empirical Analysis](https://mlsanity.com/paper/2406.14678/) (2024)
- [Patterns of Lexical Ambiguity in Contextualised Language Models](https://mlsanity.com/paper/2109.13032/) (2021)
- [Contextualized Sensorimotor Norms: multi-dimensional measures of sensorimotor strength for ambiguous English words, in context](https://mlsanity.com/paper/2203.05648/) (2022)
- [The LSCD Benchmark: a Testbed for Diachronic Word Meaning Tasks](https://mlsanity.com/paper/2404.00176/) (2024)
- [Start Making Sense(s): A Developmental Probe of Attention Specialization Using Lexical Ambiguity](https://mlsanity.com/paper/2511.21974/) (2025)
- [The Consensus Trap: Dissecting Subjectivity and the "Ground Truth" Illusion in Data Annotation](https://mlsanity.com/paper/2602.11318/) (2026)
- [Do language models capture implied discourse meanings? An investigation with exhaustivity implicatures of Korean morphology](https://mlsanity.com/paper/2405.09293/) (2024)
- [Capacity Constraints and the Multilingual Penalty for Lexical Disambiguation](https://mlsanity.com/paper/2602.05035/) (2026)

## Abstract (arXiv)

Most words are ambiguous--i.e., they convey distinct meanings in different contexts--and even the meanings of unambiguous words are context-dependent. Both phenomena present a challenge for NLP. Recently, the advent of contextualized word embeddings has led to success on tasks involving lexical ambiguity, such as Word Sense Disambiguation. However, there are few tasks that directly evaluate how well these contextualized embeddings accommodate the more continuous, dynamic nature of word meaning--particularly in a way that matches human intuitions. We introduce RAW-C, a dataset of graded, human relatedness judgments for 112 ambiguous words in context (with 672 sentence pairs total), as well as human estimates of sense dominance. The average inter-annotator agreement (assessed using a leave-one-annotator-out method) was 0.79. We then show that a measure of cosine distance, computed using contextualized embeddings from BERT and ELMo, correlates with human judgments, but that cosine distance also systematically underestimates how similar humans find uses of the same sense of a word to be, and systematically overestimates how similar humans find uses of different-sense homonyms. Finally, we propose a synthesis between psycholinguistic theories of the mental lexicon and computational models of lexical semantics.