# PixelSNAIL: An Improved Autoregressive Generative Model

arXiv: https://arxiv.org/abs/1712.09763 · HTML: https://mlsanity.com/paper/1712.09763/

## Key facts

- Cited by 137 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 Dec 2017.

## Built on by (most cited first, 30 of 137)

- [Improved Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2102.09672/) (2021)
- [Video Diffusion Models](https://mlsanity.com/paper/2204.03458/) (2022)
- [Structured Denoising Diffusion Models in Discrete State-Spaces](https://mlsanity.com/paper/2107.03006/) (2021)
- [Maximum Likelihood Training of Score-Based Diffusion Models](https://mlsanity.com/paper/2101.09258/) (2021)
- [VideoGPT: Video Generation using VQ-VAE and Transformers](https://mlsanity.com/paper/2104.10157/) (2021)
- [Vector-quantized Image Modeling with Improved VQGAN](https://mlsanity.com/paper/2110.04627/) (2021)
- [Autoregressive Image Generation without Vector Quantization](https://mlsanity.com/paper/2406.11838/) (2024)
- [Score-based Generative Modeling in Latent Space](https://mlsanity.com/paper/2106.05931/) (2021)
- [Simplified and Generalized Masked Diffusion for Discrete Data](https://mlsanity.com/paper/2406.04329/) (2024)
- [A Continuous Time Framework for Discrete Denoising Models](https://mlsanity.com/paper/2205.14987/) (2022)
- [Deep Generative Modelling: A Comparative Review of VAEs, GANs, Normalizing Flows, Energy-Based and Autoregressive Models](https://mlsanity.com/paper/2103.04922/) (2021)
- [AutoSDF: Shape Priors for 3D Completion, Reconstruction and Generation](https://mlsanity.com/paper/2203.09516/) (2022)
- [Fluid: Scaling Autoregressive Text-to-image Generative Models with Continuous Tokens](https://mlsanity.com/paper/2410.13863/) (2024)
- [Image-to-Image Translation: Methods and Applications](https://mlsanity.com/paper/2101.08629/) (2021)
- [Generating Diverse Structure for Image Inpainting With Hierarchical VQ-VAE](https://mlsanity.com/paper/2103.10022/) (2021)
- [ShapeFormer: Transformer-based Shape Completion via Sparse Representation](https://mlsanity.com/paper/2201.10326/) (2022)
- [Understanding Failures in Out-of-Distribution Detection with Deep Generative Models](https://mlsanity.com/paper/2107.06908/) (2021)
- [PixelSynth: Generating a 3D-Consistent Experience from a Single Image](https://mlsanity.com/paper/2108.05892/) (2021)
- [An Introduction to Neural Data Compression](https://mlsanity.com/paper/2202.06533/) (2022)
- [PoseGPT: Quantization-based 3D Human Motion Generation and Forecasting](https://mlsanity.com/paper/2210.10542/) (2022)
- [SQ-VAE: Variational Bayes on Discrete Representation with Self-annealed Stochastic Quantization](https://mlsanity.com/paper/2205.07547/) (2022)
- [Accelerating Auto-regressive Text-to-Image Generation with Training-free Speculative Jacobi Decoding](https://mlsanity.com/paper/2410.01699/) (2024)
- [Combiner: Full Attention Transformer with Sparse Computation Cost](https://mlsanity.com/paper/2107.05768/) (2021)
- [Predicting Video with VQVAE](https://mlsanity.com/paper/2103.01950/) (2021)
- [Meissonic: Revitalizing Masked Generative Transformers for Efficient High-Resolution Text-to-Image Synthesis](https://mlsanity.com/paper/2410.08261/) (2024)
- [Learning by Reconstruction Produces Uninformative Features For Perception](https://mlsanity.com/paper/2402.11337/) (2024)
- [Improved Techniques for Maximum Likelihood Estimation for Diffusion ODEs](https://mlsanity.com/paper/2305.03935/) (2023)
- [Densely connected normalizing flows](https://mlsanity.com/paper/2106.04627/) (2021)
- [Resampling Base Distributions of Normalizing Flows](https://mlsanity.com/paper/2110.15828/) (2021)
- [Global Context with Discrete Diffusion in Vector Quantised Modelling for Image Generation](https://mlsanity.com/paper/2112.01799/) (2021)

## Abstract (arXiv)

Autoregressive generative models consistently achieve the best results in density estimation tasks involving high dimensional data, such as images or audio. They pose density estimation as a sequence modeling task, where a recurrent neural network (RNN) models the conditional distribution over the next element conditioned on all previous elements. In this paradigm, the bottleneck is the extent to which the RNN can model long-range dependencies, and the most successful approaches rely on causal convolutions, which offer better access to earlier parts of the sequence than conventional RNNs. Taking inspiration from recent work in meta reinforcement learning, where dealing with long-range dependencies is also essential, we introduce a new generative model architecture that combines causal convolutions with self attention. In this note, we describe the resulting model and present state-of-the-art log-likelihood results on CIFAR-10 (2.85 bits per dim) and 32 × 32 ImageNet (3.80 bits per dim). Our implementation is available at https://github.com/neocxi/pixelsnail-public