# Learning to Create and Reuse Words in Open-Vocabulary Neural Language Modeling

arXiv: https://arxiv.org/abs/1704.06986 · HTML: https://mlsanity.com/paper/1704.06986/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 Apr 2017.

## Built on by (most cited first, 8 of 8)

- [CANINE: Pre-training an Efficient Tokenization-Free Encoder for Language Representation](https://mlsanity.com/paper/2103.06874/) (2021)
- [Between words and characters: A Brief History of Open-Vocabulary Modeling and Tokenization in NLP](https://mlsanity.com/paper/2112.10508/) (2021)
- [A Masked Segmental Language Model for Unsupervised Natural Language Segmentation](https://mlsanity.com/paper/2104.07829/) (2021)
- [Inducing Meaningful Units from Character Sequences with Dynamic Capacity Slot Attention](https://mlsanity.com/paper/2102.01223/) (2021)
- [Learning to Abstract with Nonparametric Variational Information Bottleneck](https://mlsanity.com/paper/2310.17284/) (2023)
- [Morphological Typology in BPE Subword Productivity and Language Modeling](https://mlsanity.com/paper/2410.23656/) (2024)
- [Revisiting Syllables in Language Modelling and their Application on Low-Resource Machine Translation](https://mlsanity.com/paper/2210.02509/) (2022)
- [Handling Open-Vocabulary Constructs in Formalizing Specifications: Retrieval-Augmented Parsing with Expert Knowledge](https://mlsanity.com/paper/2509.08808/) (2025)

## Abstract (arXiv)

Fixed-vocabulary language models fail to account for one of the most characteristic statistical facts of natural language: the frequent creation and reuse of new word types. Although character-level language models offer a partial solution in that they can create word types not attested in the training corpus, they do not capture the "bursty" distribution of such words. In this paper, we augment a hierarchical LSTM language model that generates sequences of word tokens character by character with a caching mechanism that learns to reuse previously generated words. To validate our model we construct a new open-vocabulary language modeling corpus (the Multilingual Wikipedia Corpus, MWC) from comparable Wikipedia articles in 7 typologically diverse languages and demonstrate the effectiveness of our model across this range of languages.