# Neural Lattice Language Models

arXiv: https://arxiv.org/abs/1803.05071 · HTML: https://mlsanity.com/paper/1803.05071/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 Mar 2018.

## Built on by (most cited first, 13 of 13)

- [CANINE: Pre-training an Efficient Tokenization-Free Encoder for Language Representation](https://mlsanity.com/paper/2103.06874/) (2021)
- [Between words and characters: A Brief History of Open-Vocabulary Modeling and Tokenization in NLP](https://mlsanity.com/paper/2112.10508/) (2021)
- [Turn the Combination Lock: Learnable Textual Backdoor Attacks via Word Substitution](https://mlsanity.com/paper/2106.06361/) (2021)
- [You should evaluate your language model on marginal likelihood over tokenisations](https://mlsanity.com/paper/2109.02550/) (2021)
- [The Foundations of Tokenization: Statistical and Computational Concerns](https://mlsanity.com/paper/2407.11606/) (2024)
- [Where is the signal in tokenization space?](https://mlsanity.com/paper/2408.08541/) (2024)
- [Lattice-BERT: Leveraging Multi-Granularity Representations in Chinese Pre-trained Language Models](https://mlsanity.com/paper/2104.07204/) (2021)
- [Should you marginalize over possible tokenizations?](https://mlsanity.com/paper/2306.17757/) (2023)
- [Massive-scale Decoding for Text Generation using Lattices](https://mlsanity.com/paper/2112.07660/) (2021)
- [Revisiting and Advancing Chinese Natural Language Understanding with Accelerated Heterogeneous Knowledge Pre-training](https://mlsanity.com/paper/2210.05287/) (2022)
- [LatticeGen: A Cooperative Framework which Hides Generated Text in a Lattice for Privacy-Aware Generation on Cloud](https://mlsanity.com/paper/2309.17157/) (2023)
- [Enhance Robustness of Language Models Against Variation Attack through Graph Integration](https://mlsanity.com/paper/2404.12014/) (2024)
- [Learning a Word-Level Language Model with Sentence-Level Noise Contrastive Estimation for Contextual Sentence Probability Estimation](https://mlsanity.com/paper/2103.07875/) (2021)

## Abstract (arXiv)

In this work, we propose a new language modeling paradigm that has the ability to perform both prediction and moderation of information flow at multiple granularities: neural lattice language models. These models construct a lattice of possible paths through a sentence and marginalize across this lattice to calculate sequence probabilities or optimize parameters. This approach allows us to seamlessly incorporate linguistic intuitions - including polysemy and existence of multi-word lexical items - into our language model. Experiments on multiple language modeling tasks show that English neural lattice language models that utilize polysemous embeddings are able to improve perplexity by 9.95% relative to a word-level baseline, and that a Chinese model that handles multi-character tokens is able to improve perplexity by 20.94% relative to a character-level baseline.