# Predictive Sampling with Forecasting Autoregressive Models

arXiv: https://arxiv.org/abs/2002.09928 · HTML: https://mlsanity.com/paper/2002.09928/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 Feb 2020.

## Built on by (most cited first, 10 of 10)

- [Accelerating Large Language Model Decoding with Speculative Sampling](https://mlsanity.com/paper/2302.01318/) (2023)
- [Autoregressive Diffusion Models](https://mlsanity.com/paper/2110.02037/) (2021)
- [A Review of the Gumbel-max Trick and its Extensions for Discrete Stochasticity in Machine Learning](https://mlsanity.com/paper/2110.01515/) (2021)
- [Parallel and Flexible Sampling from Autoregressive Models via Langevin Dynamics](https://mlsanity.com/paper/2105.08164/) (2021)
- [Auto-regressive Image Synthesis with Integrated Quantization](https://mlsanity.com/paper/2207.10776/) (2022)
- [DeepPCR: Parallelizing Sequential Operations in Neural Networks](https://mlsanity.com/paper/2309.16318/) (2023)
- [Parallel Sampling via Counting](https://mlsanity.com/paper/2408.09442/) (2024)
- [The Coupling Within: Flow Matching via Distilled Normalizing Flows](https://mlsanity.com/paper/2603.09014/) (2026)
- [PixelPyramids: Exact Inference Models from Lossless Image Pyramids](https://mlsanity.com/paper/2110.08787/) (2021)
- [ELF: Exact-Lipschitz Based Universal Density Approximator Flow](https://mlsanity.com/paper/2112.06997/) (2021)

## Abstract (arXiv)

Autoregressive models (ARMs) currently hold state-of-the-art performance in likelihood-based modeling of image and audio data. Generally, neural network based ARMs are designed to allow fast inference, but sampling from these models is impractically slow. In this paper, we introduce the predictive sampling algorithm: a procedure that exploits the fast inference property of ARMs in order to speed up sampling, while keeping the model intact. We propose two variations of predictive sampling, namely sampling with ARM fixed-point iteration and learned forecasting modules. Their effectiveness is demonstrated in two settings: i) explicit likelihood modeling on binary MNIST, SVHN and CIFAR10, and ii) discrete latent modeling in an autoencoder trained on SVHN, CIFAR10 and Imagenet32. Empirically, we show considerable improvements over baselines in number of ARM inference calls and sampling speed.