# A Heavy-Tailed Algebra for Probabilistic Programming

arXiv: https://arxiv.org/abs/2306.09262 · HTML: https://mlsanity.com/paper/2306.09262/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 59 references that are in the map.
- First posted on arXiv 15 Jun 2023.

## Builds on (most cited first, 19 of 19)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Normalizing Flows for Probabilistic Modeling and Inference](https://mlsanity.com/paper/1912.02762/) (2019)
- [Neural Spline Flows](https://mlsanity.com/paper/1906.04032/) (2019)
- [Importance Weighted Autoencoders](https://mlsanity.com/paper/1509.00519/) (2015)
- [Pyro: Deep Universal Probabilistic Programming](https://mlsanity.com/paper/1810.09538/) (2018)
- [A Tail-Index Analysis of Stochastic Gradient Noise in Deep Neural Networks](https://mlsanity.com/paper/1901.06053/) (2019)
- [Attend, Infer, Repeat: Fast Scene Understanding with Generative Models](https://mlsanity.com/paper/1603.08575/) (2016)
- [Automatic Differentiation Variational Inference](https://mlsanity.com/paper/1603.00788/) (2016)
- [The Heavy-Tail Phenomenon in SGD](https://mlsanity.com/paper/2006.04740/) (2020)
- [Learning Disentangled Representations with Semi-Supervised Deep Generative Models](https://mlsanity.com/paper/1706.00400/) (2017)
- [Yes, but Did It Work?: Evaluating Variational Inference](https://mlsanity.com/paper/1802.02538/) (2018)
- [Pareto Smoothed Importance Sampling](https://mlsanity.com/paper/1507.02646/) (2015)
- [Disintegration and Bayesian Inversion via String Diagrams](https://mlsanity.com/paper/1709.00322/) (2017)
- [Variational Inference with Tail-adaptive f-Divergence](https://mlsanity.com/paper/1810.11943/) (2018)
- [Simple, Distributed, and Accelerated Probabilistic Programming](https://mlsanity.com/paper/1811.02091/) (2018)
- [Design and Implementation of Probabilistic Programming Language Anglican](https://mlsanity.com/paper/1608.05263/) (2016)
- [Fat-Tailed Variational Inference with Anisotropic Tail Adaptive Flows](https://mlsanity.com/paper/2205.07918/) (2022)
- [Towards Verified Stochastic Variational Inference for Probabilistic Programs](https://mlsanity.com/paper/1907.08827/) (2019)
- [Static Analysis for Probabilistic Programs](https://mlsanity.com/paper/1909.05076/) (2019)

## Built on by (most cited first, 2 of 2)

- [Flexible Tails for Normalizing Flows](https://mlsanity.com/paper/2406.16971/) (2024)
- [Learning Physics for Unveiling Hidden Earthquake Ground Motions via Conditional Generative Modeling](https://mlsanity.com/paper/2407.15089/) (2024)

## Abstract (arXiv)

Despite the successes of probabilistic models based on passing noise through neural networks, recent work has identified that such methods often fail to capture tail behavior accurately, unless the tails of the base distribution are appropriately calibrated. To overcome this deficiency, we propose a systematic approach for analyzing the tails of random variables, and we illustrate how this approach can be used during the static analysis (before drawing samples) pass of a probabilistic programming language compiler. To characterize how the tails change under various operations, we develop an algebra which acts on a three-parameter family of tail asymptotics and which is based on the generalized Gamma distribution. Our algebraic operations are closed under addition and multiplication; they are capable of distinguishing sub-Gaussians with differing scales; and they handle ratios sufficiently well to reproduce the tails of most important statistical distributions directly from their definitions. Our empirical results confirm that inference algorithms that leverage our heavy-tailed algebra attain superior performance across a number of density modeling and variational inference tasks.