# Extrapolation in NLP

arXiv: https://arxiv.org/abs/1805.06648 · HTML: https://mlsanity.com/paper/1805.06648/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 17 May 2018.

## Built on by (most cited first, 2 of 2)

- [Length Extrapolation of Transformers: A Survey from the Perspective of Positional Encoding](https://mlsanity.com/paper/2312.17044/) (2023)
- [Relational Weight Priors in Neural Networks for Abstract Pattern Learning and Language Modelling](https://mlsanity.com/paper/2103.06198/) (2021)

## Abstract (arXiv)

We argue that extrapolation to examples outside the training space will often be easier for models that capture global structures, rather than just maximise their local fit to the training data. We show that this is true for two popular models: the Decomposable Attention Model and word2vec.