# High-risk learning: acquiring new word vectors from tiny data

arXiv: https://arxiv.org/abs/1707.06556 · HTML: https://mlsanity.com/paper/1707.06556/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 20 Jul 2017.

## Built on by (most cited first, 13 of 13)

- [An Empirical Survey of Data Augmentation for Limited Data Learning in NLP](https://mlsanity.com/paper/2106.07499/) (2021)
- [Dynamic Language Models for Continuously Evolving Content](https://mlsanity.com/paper/2106.06297/) (2021)
- [Recent Trends in Named Entity Recognition (NER)](https://mlsanity.com/paper/2101.11420/) (2021)
- [The Impact of Word Splitting on the Semantic Content of Contextualized Word Representations](https://mlsanity.com/paper/2402.14616/) (2024)
- [Lacking the embedding of a word? Look it up into a traditional dictionary](https://mlsanity.com/paper/2109.11763/) (2021)
- [Rapid Word Learning Through Meta In-Context Learning](https://mlsanity.com/paper/2502.14791/) (2025)
- [CoLLEGe: Concept Embedding Generation for Large Language Models](https://mlsanity.com/paper/2403.15362/) (2024)
- [Measuring Contextual Informativeness in Child-Directed Text](https://mlsanity.com/paper/2412.17427/) (2024)
- [Trajectory-Based Meta-Learning for Out-Of-Vocabulary Word Embedding Learning](https://mlsanity.com/paper/2102.12266/) (2021)
- [Novel Aficionados and Doppelg"angers: a referential task for semantic representations of individual entities](https://mlsanity.com/paper/2104.10270/) (2021)
- [An Attention-Based Model for Predicting Contextual Informativeness and Curriculum Learning Applications](https://mlsanity.com/paper/2204.09885/) (2022)
- [Using dependency parsing for few-shot learning in distributional semantics](https://mlsanity.com/paper/2205.06168/) (2022)
- [Dynamics of Meaning: Towards the Evaluation of Diachronic Semantic Change in Sinhala](https://mlsanity.com/paper/2609.08609/) (2026)

## Abstract (arXiv)

Distributional semantics models are known to struggle with small data. It is generally accepted that in order to learn 'a good vector' for a word, a model must have sufficient examples of its usage. This contradicts the fact that humans can guess the meaning of a word from a few occurrences only. In this paper, we show that a neural language model such as Word2Vec only necessitates minor modifications to its standard architecture to learn new terms from tiny data, using background knowledge from a previously learnt semantic space. We test our model on word definitions and on a nonce task involving 2-6 sentences' worth of context, showing a large increase in performance over state-of-the-art models on the definitional task.