# Deep Generative Stochastic Networks Trainable by Backprop

arXiv: https://arxiv.org/abs/1306.1091 · HTML: https://mlsanity.com/paper/1306.1091/

## Key facts

- Cited by 36 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 5 Jun 2013.

## Built on by (most cited first, 30 of 36)

- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Score-Based Generative Modeling with Critically-Damped Langevin Diffusion](https://mlsanity.com/paper/2112.07068/) (2021)
- [Versatile Diffusion: Text, Images and Variations All in One Diffusion Model](https://mlsanity.com/paper/2211.08332/) (2022)
- [Smooth Diffusion: Crafting Smooth Latent Spaces in Diffusion Models](https://mlsanity.com/paper/2312.04410/) (2023)
- [The Predictive Forward-Forward Algorithm](https://mlsanity.com/paper/2301.01452/) (2023)
- [On quantum backpropagation, information reuse, and cheating measurement collapse](https://mlsanity.com/paper/2305.13362/) (2023)
- [Dynamic Dual-Output Diffusion Models](https://mlsanity.com/paper/2203.04304/) (2022)
- [Posterior samples of source galaxies in strong gravitational lenses with score-based priors](https://mlsanity.com/paper/2211.03812/) (2022)
- [Fast Sampling via Discrete Non-Markov Diffusion Models with Predetermined Transition Time](https://mlsanity.com/paper/2312.09193/) (2023)
- [Improved Autoregressive Modeling with Distribution Smoothing](https://mlsanity.com/paper/2103.15089/) (2021)
- [A Study of the Generalizability of Self-Supervised Representations](https://mlsanity.com/paper/2109.09150/) (2021)
- [Generative Diffusion Models for Radio Wireless Channel Modelling and Sampling](https://mlsanity.com/paper/2308.05583/) (2023)
- [Probing BERT's priors with serial reproduction chains](https://mlsanity.com/paper/2202.12226/) (2022)
- [Counterfactual Generative Modeling with Variational Causal Inference](https://mlsanity.com/paper/2410.12730/) (2024)
- [Revealing the Distributional Vulnerability of Discriminators by Implicit Generators](https://mlsanity.com/paper/2108.09976/) (2021)
- [DiffSketching: Sketch Control Image Synthesis with Diffusion Models](https://mlsanity.com/paper/2305.18812/) (2023)
- [Deriving Language Models from Masked Language Models](https://mlsanity.com/paper/2305.15501/) (2023)
- [Image generation with shortest path diffusion](https://mlsanity.com/paper/2306.00501/) (2023)
- [Enhancing Mechanical Metamodels with a Generative Model-Based Augmented Training Dataset](https://mlsanity.com/paper/2203.04183/) (2022)
- [Bring Metric Functions into Diffusion Models](https://mlsanity.com/paper/2401.02414/) (2024)
- [AdvNF: Reducing Mode Collapse in Conditional Normalising Flows using Adversarial Learning](https://mlsanity.com/paper/2401.15948/) (2024)
- [Learning from demonstration using products of experts: applications to manipulation and task prioritization](https://mlsanity.com/paper/2010.03505/) (2020)
- [On the Generative Utility of Cyclic Conditionals](https://mlsanity.com/paper/2106.15962/) (2021)
- [Variational Causal Inference](https://mlsanity.com/paper/2209.05935/) (2022)
- [A Diffusion-based Method for Multi-turn Compositional Image Generation](https://mlsanity.com/paper/2304.02192/) (2023)
- [One-Line-of-Code Data Mollification Improves Optimization of Likelihood-based Generative Models](https://mlsanity.com/paper/2305.18900/) (2023)
- [F-Divergences and Cost Function Locality in Generative Modelling with Quantum Circuits](https://mlsanity.com/paper/2110.04253/) (2021)
- [Taming Normalizing Flows](https://mlsanity.com/paper/2211.16488/) (2022)
- [A Generic Self-Supervised Framework of Learning Invariant Discriminative Features](https://mlsanity.com/paper/2202.06914/) (2022)
- [fairDMS: Rapid Model Training by Data and Model Reuse](https://mlsanity.com/paper/2204.09805/) (2022)

## Abstract (arXiv)

We introduce a novel training principle for probabilistic models that is an alternative to maximum likelihood. The proposed Generative Stochastic Networks (GSN) framework is based on learning the transition operator of a Markov chain whose stationary distribution estimates the data distribution. The transition distribution of the Markov chain is conditional on the previous state, generally involving a small move, so this conditional distribution has fewer dominant modes, being unimodal in the limit of small moves. Thus, it is easier to learn because it is easier to approximate its partition function, more like learning to perform supervised function approximation, with gradients that can be obtained by backprop. We provide theorems that generalize recent work on the probabilistic interpretation of denoising autoencoders and obtain along the way an interesting justification for dependency networks and generalized pseudolikelihood, along with a definition of an appropriate joint distribution and sampling mechanism even when the conditionals are not consistent. GSNs can be used with missing inputs and can be used to sample subsets of variables given the rest. We validate these theoretical results with experiments on two image datasets using an architecture that mimics the Deep Boltzmann Machine Gibbs sampler but allows training to proceed with simple backprop, without the need for layerwise pretraining.