# Trajectory-Based Meta-Learning for Out-Of-Vocabulary Word Embedding Learning

arXiv: https://arxiv.org/abs/2102.12266 · HTML: https://mlsanity.com/paper/2102.12266/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 21 of its 29 references that are in the map.
- First posted on arXiv 24 Feb 2021.

## Builds on (most cited first, 21 of 21)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://mlsanity.com/paper/1703.03400/) (2017)
- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [Pointer Sentinel Mixture Models](https://mlsanity.com/paper/1609.07843/) (2016)
- [Prototypical Networks for Few-shot Learning](https://mlsanity.com/paper/1703.05175/) (2017)
- [Matching Networks for One Shot Learning](https://mlsanity.com/paper/1606.04080/) (2016)
- [Enriching Word Vectors with Subword Information](https://mlsanity.com/paper/1607.04606/) (2016)
- [On First-Order Meta-Learning Algorithms](https://mlsanity.com/paper/1803.02999/) (2018)
- [Neural Architectures for Named Entity Recognition](https://mlsanity.com/paper/1603.01360/) (2016)
- [How to train your MAML](https://mlsanity.com/paper/1810.09502/) (2018)
- [Character-Aware Neural Language Models](https://mlsanity.com/paper/1508.06615/) (2015)
- [Generalized Inner Loop Meta-Learning](https://mlsanity.com/paper/1910.01727/) (2019)
- [Understanding Short-Horizon Bias in Stochastic Meta-Optimization](https://mlsanity.com/paper/1803.02021/) (2018)
- [Mimicking Word Embeddings using Subword RNNs](https://mlsanity.com/paper/1707.06961/) (2017)
- [Few-Shot Representation Learning for Out-Of-Vocabulary Words](https://mlsanity.com/paper/1907.00505/) (2019)
- [Transferring Knowledge across Learning Processes](https://mlsanity.com/paper/1812.01054/) (2018)
- [A La Carte Embedding: Cheap but Effective Induction of Semantic Feature Vectors](https://mlsanity.com/paper/1805.05388/) (2018)
- [Contextual Embeddings: When Are They Worth It?](https://mlsanity.com/paper/2005.09117/) (2020)
- [High-risk learning: acquiring new word vectors from tiny data](https://mlsanity.com/paper/1707.06556/) (2017)
- [Predicting and interpreting embeddings for out of vocabulary words in downstream tasks](https://mlsanity.com/paper/1903.00724/) (2019)
- [Domain Adapted Word Embeddings for Improved Sentiment Classification](https://mlsanity.com/paper/1805.04576/) (2018)

## Abstract (arXiv)

Word embedding learning methods require a large number of occurrences of a word to accurately learn its embedding. However, out-of-vocabulary (OOV) words which do not appear in the training corpus emerge frequently in the smaller downstream data. Recent work formulated OOV embedding learning as a few-shot regression problem and demonstrated that meta-learning can improve results obtained. However, the algorithm used, model-agnostic meta-learning (MAML) is known to be unstable and perform worse when a large number of gradient steps are used for parameter updates. In this work, we propose the use of Leap, a meta-learning algorithm which leverages the entire trajectory of the learning process instead of just the beginning and the end points, and thus ameliorates these two issues. In our experiments on a benchmark OOV embedding learning dataset and in an extrinsic evaluation, Leap performs comparably or better than MAML. We go on to examine which contexts are most beneficial to learn an OOV embedding from, and propose that the choice of contexts may matter more than the meta-learning employed.