# Deep AutoRegressive Networks

arXiv: https://arxiv.org/abs/1310.8499 · HTML: https://mlsanity.com/paper/1310.8499/

## Key facts

- Cited by 61 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 31 Oct 2013.

## Built on by (most cited first, 30 of 61)

- [Autoregressive Image Generation without Vector Quantization](https://mlsanity.com/paper/2406.11838/) (2024)
- [Hunyuan3D 2.0: Scaling Diffusion Models for High Resolution Textured 3D Assets Generation](https://mlsanity.com/paper/2501.12202/) (2025)
- [Fluid: Scaling Autoregressive Text-to-image Generative Models with Continuous Tokens](https://mlsanity.com/paper/2410.13863/) (2024)
- [UNIT-DDPM: UNpaired Image Translation with Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2104.05358/) (2021)
- [Randomized Autoregressive Visual Generation](https://mlsanity.com/paper/2411.00776/) (2024)
- [Multimodal Image Synthesis and Editing: The Generative AI Era](https://mlsanity.com/paper/2112.13592/) (2021)
- [Attention, please! A survey of Neural Attention Models in Deep Learning](https://mlsanity.com/paper/2103.16775/) (2021)
- [Unified Autoregressive Visual Generation and Understanding with Continuous Tokens](https://mlsanity.com/paper/2503.13436/) (2025)
- [Community Forensics: Using Thousands of Generators to Train Fake Image Detectors](https://mlsanity.com/paper/2411.04125/) (2024)
- [Democratizing Text-to-Image Masked Generative Models with Compact Text-Aware One-Dimensional Tokens](https://mlsanity.com/paper/2501.07730/) (2025)
- [Regularized Vector Quantization for Tokenized Image Synthesis](https://mlsanity.com/paper/2303.06424/) (2023)
- [HDMapGen: A Hierarchical Graph Generative Model of High Definition Maps](https://mlsanity.com/paper/2106.14880/) (2021)
- [Counting Ability of Large Language Models and Impact of Tokenization](https://mlsanity.com/paper/2410.19730/) (2024)
- [Bridging Discrete and Backpropagation: Straight-Through and Beyond](https://mlsanity.com/paper/2304.08612/) (2023)
- [Probabilistic Mixture-of-Experts for Efficient Deep Reinforcement Learning](https://mlsanity.com/paper/2104.09122/) (2021)
- [Sampling with flows, diffusion and autoregressive neural networks: A spin-glass perspective](https://mlsanity.com/paper/2308.14085/) (2023)
- [Feature Shift Detection: Localizing Which Features Have Shifted via Conditional Distribution Tests](https://mlsanity.com/paper/2107.06929/) (2021)
- [MedDiff: Generating Electronic Health Records using Accelerated Denoising Diffusion Model](https://mlsanity.com/paper/2302.04355/) (2023)
- [Catch Missing Details: Image Reconstruction with Frequency Augmented Variational Autoencoder](https://mlsanity.com/paper/2305.02541/) (2023)
- [Towards Long-Term Time-Series Forecasting: Feature, Pattern, and Distribution](https://mlsanity.com/paper/2301.02068/) (2023)
- [APEX: An Extensible and Dynamism-Aware Simulator for Automated Parallel Execution in LLM Serving](https://mlsanity.com/paper/2411.17651/) (2024)
- [Differentiable Sampling of Categorical Distributions Using the CatLog-Derivative Trick](https://mlsanity.com/paper/2311.12569/) (2023)
- [Denoising with a Joint-Embedding Predictive Architecture](https://mlsanity.com/paper/2410.03755/) (2024)
- [Tensor networks and efficient descriptions of classical data](https://mlsanity.com/paper/2103.06872/) (2021)
- [Learning from Predictions: Fusing Training and Autoregressive Inference for Long-Term Spatiotemporal Forecasts](https://mlsanity.com/paper/2302.11101/) (2023)
- [LazyMAR: Accelerating Masked Autoregressive Models via Feature Caching](https://mlsanity.com/paper/2503.12450/) (2025)
- [Continuous Visual Autoregressive Generation via Score Maximization](https://mlsanity.com/paper/2505.07812/) (2025)
- [The autoregressive neural network architecture of the Boltzmann distribution of pairwise interacting spins systems](https://mlsanity.com/paper/2302.08347/) (2023)
- [Autoregressive Images Watermarking through Lexical Biasing: An Approach Resistant to Regeneration Attack](https://mlsanity.com/paper/2506.01011/) (2025)
- [Bridging the Training-Inference Gap in LLMs by Leveraging Self-Generated Tokens](https://mlsanity.com/paper/2410.14655/) (2024)

## Abstract (arXiv)

We introduce a deep, generative autoencoder capable of learning hierarchies of distributed representations from data. Successive deep stochastic hidden layers are equipped with autoregressive connections, which enable the model to be sampled from quickly and exactly via ancestral sampling. We derive an efficient approximate parameter estimation method based on the minimum description length (MDL) principle, which can be seen as maximising a variational lower bound on the log-likelihood, with a feedforward neural network implementing approximate inference. We demonstrate state-of-the-art generative performance on a number of classic data sets: several UCI data sets, MNIST and Atari 2600 games.