# Tensor Monte Carlo: particle methods for the GPU era

arXiv: https://arxiv.org/abs/1806.08593 · HTML: https://mlsanity.com/paper/1806.08593/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 22 Jun 2018.

## Built on by (most cited first, 8 of 8)

- [Differentiable Particle Filtering via Entropy-Regularized Optimal Transport](https://mlsanity.com/paper/2102.07850/) (2021)
- [Data augmentation in Bayesian neural networks and the cold posterior effect](https://mlsanity.com/paper/2106.05586/) (2021)
- [Multiple Importance Sampling ELBO and Deep Ensembles of Variational Approximations](https://mlsanity.com/paper/2202.10951/) (2022)
- [Variational Marginal Particle Filters](https://mlsanity.com/paper/2109.15134/) (2021)
- [PASOA- PArticle baSed Bayesian Optimal Adaptive design](https://mlsanity.com/paper/2402.07160/) (2024)
- [De-Sequentialized Monte Carlo: a parallel-in-time particle smoother](https://mlsanity.com/paper/2202.02264/) (2022)
- [Massively Parallel Reweighted Wake-Sleep](https://mlsanity.com/paper/2305.11022/) (2023)
- [Massively Parallel Expectation Maximization For Approximate Posteriors](https://mlsanity.com/paper/2503.08264/) (2025)

## Abstract (arXiv)

Multi-sample, importance-weighted variational autoencoders (IWAE) give tighter bounds and more accurate uncertainty estimates than variational autoencoders (VAE) trained with a standard single-sample objective. However, IWAEs scale poorly: as the latent dimensionality grows, they require exponentially many samples to retain the benefits of importance weighting. While sequential Monte-Carlo (SMC) can address this problem, it is prohibitively slow because the resampling step imposes sequential structure which cannot be parallelised, and moreover, resampling is non-differentiable which is problematic when learning approximate posteriors. To address these issues, we developed tensor Monte-Carlo (TMC) which gives exponentially many importance samples by separately drawing K samples for each of the n latent variables, then averaging over all Kn possible combinations. While the sum over exponentially many terms might seem to be intractable, in many cases it can be computed efficiently as a series of tensor inner-products. We show that TMC is superior to IWAE on a generative model with multiple stochastic layers trained on the MNIST handwritten digit database, and we show that TMC can be combined with standard variance reduction techniques.