# Autoregressive Models: What Are They Good For?

arXiv: https://arxiv.org/abs/1910.07737 · HTML: https://mlsanity.com/paper/1910.07737/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 17 Oct 2019.

## Built on by (most cited first, 4 of 4)

- [Plasma Surrogate Modelling using Fourier Neural Operators](https://mlsanity.com/paper/2311.05967/) (2023)
- [Continuous Latent Diffusion Language Model](https://mlsanity.com/paper/2605.06548/) (2026)
- [An Evaluation of Large Language Models on Text Summarization Tasks Using Prompt Engineering Techniques](https://mlsanity.com/paper/2507.05123/) (2025)
- [Synthetic ECG Generation for Data Augmentation and Transfer Learning in Arrhythmia Classification](https://mlsanity.com/paper/2411.18456/) (2024)

## Abstract (arXiv)

Autoregressive (AR) models have become a popular tool for unsupervised learning, achieving state-of-the-art log likelihood estimates. We investigate the use of AR models as density estimators in two settings -- as a learning signal for image translation, and as an outlier detector -- and find that these density estimates are much less reliable than previously thought. We examine the underlying optimization issues from both an empirical and theoretical perspective, and provide a toy example that illustrates the problem. Overwhelmingly, we find that density estimates do not correlate with perceptual quality and are unhelpful for downstream tasks.