# The Importance of Generation Order in Language Modeling

arXiv: https://arxiv.org/abs/1808.07910 · HTML: https://mlsanity.com/paper/1808.07910/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 Aug 2018.

## Built on by (most cited first, 9 of 9)

- [Learning to Complete Code with Sketches](https://mlsanity.com/paper/2106.10158/) (2021)
- [Arrows of Time for Large Language Models](https://mlsanity.com/paper/2401.17505/) (2024)
- [Discovering Non-monotonic Autoregressive Orderings with Variational Inference](https://mlsanity.com/paper/2110.15797/) (2021)
- [Parrot Mind: Towards Explaining the Complex Task Reasoning of Pretrained Large Language Models with Template-Content Structure](https://mlsanity.com/paper/2310.05452/) (2023)
- [Improving Discrete Diffusion Models via Structured Preferential Generation](https://mlsanity.com/paper/2405.17889/) (2024)
- [Towards More Efficient Insertion Transformer with Fractional Positional Encoding](https://mlsanity.com/paper/2112.06295/) (2021)
- [Distilling Specialized Orders for Visual Generation](https://mlsanity.com/paper/2504.17069/) (2025)
- [New Algorithmic Directions in Optimal Transport and Applications for Product Spaces](https://mlsanity.com/paper/2509.21502/) (2025)
- [Characterizing Cultural Localization in AI-Generated Stories](https://mlsanity.com/paper/2606.14626/) (2026)

## Abstract (arXiv)

Neural language models are a critical component of state-of-the-art systems for machine translation, summarization, audio transcription, and other tasks. These language models are almost universally autoregressive in nature, generating sentences one token at a time from left to right. This paper studies the influence of token generation order on model quality via a novel two-pass language model that produces partially-filled sentence "templates" and then fills in missing tokens. We compare various strategies for structuring these two passes and observe a surprisingly large variation in model quality. We find the most effective strategy generates function words in the first pass followed by content words in the second. We believe these experimental results justify a more extensive investigation of generation order for neural language models.