# Markov Chain Decoders Overcome the Heavy-Tail Limitations of Lipschitz Generative Models

arXiv: https://arxiv.org/abs/2605.18931 · HTML: https://mlsanity.com/paper/2605.18931/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 7 of its 31 references that are in the map.
- First posted on arXiv 18 May 2026.

## Builds on (most cited first, 7 of 7)

- [An Introduction to Variational Autoencoders](https://mlsanity.com/paper/1906.02691/) (2019)
- [Tutorial on Variational Autoencoders](https://mlsanity.com/paper/1606.05908/) (2016)
- [Variational Lossy Autoencoder](https://mlsanity.com/paper/1611.02731/) (2016)
- [ExGAN: Adversarial Generation of Extreme Samples](https://mlsanity.com/paper/2009.08454/) (2020)
- [t3-Variational Autoencoder: Learning Heavy-tailed Data with Student's t and Power Divergence](https://mlsanity.com/paper/2312.01133/) (2023)
- [A VAE Approach to Sample Multivariate Extremes](https://mlsanity.com/paper/2306.10987/) (2023)
- [Approximating Heavy-Tailed Distributions with a Mixture of Bernstein Phase-Type and Hyperexponential Models](https://mlsanity.com/paper/2510.26524/) (2025)

## Abstract (arXiv)

Heavy-tailed distributions are prevalent in performance evaluation, network traffic, and risk modeling. This behavior poses a fundamental challenge for modern deep generative models. Standard Variational Autoencoders (VAEs) employ Gaussian decoder likelihoods and Lipschitz-constrained neural networks, a combination that is structurally incapable of producing heavy-tailed outputs: the Gaussian tail decays exponentially, and Lipschitz continuity prevents the decoder from amplifying rare events from the latent space input to sufficiently overcome this decay. We provide both a theoretical characterization of this limitation and a controlled empirical demonstration using synthetic Pareto data across a grid of tail indices α ∈ 2, 3, 5, 30 and dimensions d ∈ 1, 5, 10. As a solution, we replace the Gaussian decoder with a Phase-Type (PH) distribution based on Markov chains, while keeping the encoder, latent space, and training procedure identical. PH distributions allow for arbitrarily precise approximations of any positive-valued distributions, including heavy-tailed families. Experiments showed that the PH-based model reduces tail Kolmogorov-Smirnov distance by up to x6 and extreme quantile error by up to x10 compared to the Gaussian baseline for heavy-tailed data. These results demonstrate that integrating Markov chain-based distributions into the decoder of a generative model institutes a principled and practically effective solution to the heavy-tail generation problem.