# Rapid Word Learning Through Meta In-Context Learning

arXiv: https://arxiv.org/abs/2502.14791 · HTML: https://mlsanity.com/paper/2502.14791/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 65 references that are in the map.
- First posted on arXiv 20 Feb 2025.

## Builds on (most cited first, 30 of 32)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference](https://mlsanity.com/paper/1704.05426/) (2017)
- [Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling](https://mlsanity.com/paper/2304.01373/) (2023)
- [BLiMP: The Benchmark of Linguistic Minimal Pairs for English](https://mlsanity.com/paper/1912.00582/) (2019)
- [MetaICL: Learning to Learn In Context](https://mlsanity.com/paper/2110.15943/) (2021)
- [Meta-learning via Language Model In-context Tuning](https://mlsanity.com/paper/2110.07814/) (2021)
- [Large Language Models are Inconsistent and Biased Evaluators](https://mlsanity.com/paper/2405.01724/) (2024)
- [Between words and characters: A Brief History of Open-Vocabulary Modeling and Tokenization in NLP](https://mlsanity.com/paper/2112.10508/) (2021)
- [Analyzing Uncertainty in Neural Machine Translation](https://mlsanity.com/paper/1803.00047/) (2018)
- [How Can We Accelerate Progress Towards Human-like Linguistic Generalization?](https://mlsanity.com/paper/2005.00955/) (2020)
- [Frequency Effects on Syntactic Rule Learning in Transformers](https://mlsanity.com/paper/2109.07020/) (2021)
- [ICLR: In-Context Learning of Representations](https://mlsanity.com/paper/2501.00070/) (2024)
- [Rare Words: A Major Problem for Contextualized Embeddings And How to Fix it by Attentive Mimicking](https://mlsanity.com/paper/1904.06707/) (2019)
- [Conditional Generators of Words Definitions](https://mlsanity.com/paper/1806.10090/) (2018)
- [Learning to Understand Phrases by Embedding the Dictionary](https://mlsanity.com/paper/1504.00548/) (2015)
- [Few-Shot Representation Learning for Out-Of-Vocabulary Words](https://mlsanity.com/paper/1907.00505/) (2019)
- [Uncontrolled Lexical Exposure Leads to Overestimation of Compositional Generalization in Pretrained Models](https://mlsanity.com/paper/2212.10769/) (2022)
- [Language model acceptability judgements are not always robust to context](https://mlsanity.com/paper/2212.08979/) (2022)
- [In-context Learning Generalizes, But Not Always Robustly: The Case of Syntax](https://mlsanity.com/paper/2311.07811/) (2023)
- [Interpretable Word Sense Representations via Definition Generation: The Case of Semantic Change Analysis](https://mlsanity.com/paper/2305.11993/) (2023)
- [A La Carte Embedding: Cheap but Effective Induction of Semantic Feature Vectors](https://mlsanity.com/paper/1805.05388/) (2018)
- [WinoDict: Probing language models for in-context word acquisition](https://mlsanity.com/paper/2209.12153/) (2022)
- [Dual Process Learning: Controlling Use of In-Context vs. In-Weights Strategies with Weight Forgetting](https://mlsanity.com/paper/2406.00053/) (2024)
- [High-risk learning: acquiring new word vectors from tiny data](https://mlsanity.com/paper/1707.06556/) (2017)
- [How Reliable Are Automatic Evaluation Methods for Instruction-Tuned LLMs?](https://mlsanity.com/paper/2402.10770/) (2024)
- [Learning Semantic Representations for Novel Words: Leveraging Both Form and Context](https://mlsanity.com/paper/1811.03866/) (2018)

## Built on by (most cited first, 4 of 4)

- [Whither symbols in the era of advanced neural networks?](https://mlsanity.com/paper/2508.05776/) (2025)
- [Would you still call this Dax? Novel Visual References in VLMs and Humans](https://mlsanity.com/paper/2606.05409/) (2026)
- [Children, but not language models, show accelerating returns in word learning](https://mlsanity.com/paper/2608.17120/) (2026)
- [(V)LMs generalize beyond surface co-occurrence: Evidence from cross-modal number agreement](https://mlsanity.com/paper/2609.00443/) (2026)

## Abstract (arXiv)

Humans can quickly learn a new word from a few illustrative examples, and then systematically and flexibly use it in novel contexts. Yet the abilities of current language models for few-shot word learning, and methods for improving these abilities, are underexplored. In this study, we introduce a novel method, Meta-training for IN-context learNing Of Words (Minnow). This method trains language models to generate new examples of a word's usage given a few in-context examples, using a special placeholder token to represent the new word. This training is repeated on many new words to develop a general word-learning ability. We find that training models from scratch with Minnow on human-scale child-directed language enables strong few-shot word learning, comparable to a large language model (LLM) pre-trained on orders of magnitude more data. Furthermore, through discriminative and generative evaluations, we demonstrate that finetuning pre-trained LLMs with Minnow improves their ability to discriminate between new words, identify syntactic categories of new words, and generate reasonable new usages and definitions for new words, based on one or a few in-context examples. These findings highlight the data efficiency of Minnow and its potential to improve language model performance in word learning tasks.