# Stochastic Backpropagation through Mixture Density Distributions

arXiv: https://arxiv.org/abs/1607.05690 · HTML: https://mlsanity.com/paper/1607.05690/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 19 Jul 2016.

## Built on by (most cited first, 14 of 14)

- [Differentiable Particle Filtering via Entropy-Regularized Optimal Transport](https://mlsanity.com/paper/2102.07850/) (2021)
- [Self-Conditioned Probabilistic Learning of Video Rescaling](https://mlsanity.com/paper/2107.11639/) (2021)
- [Differentiable Particle Filtering without Modifying the Forward Pass](https://mlsanity.com/paper/2106.10314/) (2021)
- [Variational Distillation of Diffusion Policies into Mixture of Experts](https://mlsanity.com/paper/2406.12538/) (2024)
- [Particle Semi-Implicit Variational Inference](https://mlsanity.com/paper/2407.00649/) (2024)
- [Variational Marginal Particle Filters](https://mlsanity.com/paper/2109.15134/) (2021)
- [Preventing Model Collapse in Gaussian Process Latent Variable Models](https://mlsanity.com/paper/2404.01697/) (2024)
- [TRACE: A Differentiable Approach to Line-level Stroke Recovery for Offline Handwritten Text](https://mlsanity.com/paper/2105.11559/) (2021)
- [Training Latent Variable Models with Auto-encoding Variational Bayes: A Tutorial](https://mlsanity.com/paper/2208.07818/) (2022)
- [Differentiable and Stable Long-Range Tracking of Multiple Posterior Modes](https://mlsanity.com/paper/2404.08789/) (2024)
- [Deep Variational Sequential Monte Carlo for High-Dimensional Observations](https://mlsanity.com/paper/2501.05982/) (2025)
- [Continuous Autoregressive Modeling with Stochastic Monotonic Alignment for Speech Synthesis](https://mlsanity.com/paper/2502.01084/) (2025)
- [Multi-View Oriented GPLVM: Expressiveness and Efficiency](https://mlsanity.com/paper/2502.08253/) (2025)
- [Robust scalable initialization for Bayesian variational inference with multi-modal Laplace approximations](https://mlsanity.com/paper/2307.06424/) (2023)

## Abstract (arXiv)

The ability to backpropagate stochastic gradients through continuous latent distributions has been crucial to the emergence of variational autoencoders and stochastic gradient variational Bayes. The key ingredient is an unbiased and low-variance way of estimating gradients with respect to distribution parameters from gradients evaluated at distribution samples. The "reparameterization trick" provides a class of transforms yielding such estimators for many continuous distributions, including the Gaussian and other members of the location-scale family. However the trick does not readily extend to mixture density models, due to the difficulty of reparameterizing the discrete distribution over mixture weights. This report describes an alternative transform, applicable to any continuous multivariate distribution with a differentiable density function from which samples can be drawn, and uses it to derive an unbiased estimator for mixture density weight derivatives. Combined with the reparameterization trick applied to the individual mixture components, this estimator makes it straightforward to train variational autoencoders with mixture-distributed latent variables, or to perform stochastic variational inference with a mixture density variational posterior.