# Unsupervised tree boosting for learning probability distributions

arXiv: https://arxiv.org/abs/2101.11083 · HTML: https://mlsanity.com/paper/2101.11083/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 36 references that are in the map.
- First posted on arXiv 26 Jan 2021.

## Builds on (most cited first, 10 of 10)

- [Density estimation using Real NVP](https://mlsanity.com/paper/1605.08803/) (2016)
- [Normalizing Flows for Probabilistic Modeling and Inference](https://mlsanity.com/paper/1912.02762/) (2019)
- [Masked Autoregressive Flow for Density Estimation](https://mlsanity.com/paper/1705.07057/) (2017)
- [MADE: Masked Autoencoder for Distribution Estimation](https://mlsanity.com/paper/1502.03509/) (2015)
- [Neural Autoregressive Flows](https://mlsanity.com/paper/1804.00779/) (2018)
- [Sum-of-Squares Polynomial Flow](https://mlsanity.com/paper/1905.02325/) (2019)
- [The Expressive Power of a Class of Normalizing Flow Models](https://mlsanity.com/paper/2006.00392/) (2020)
- [Neural Density Estimation and Likelihood-free Inference](https://mlsanity.com/paper/1910.13233/) (2019)
- [Stochastic tree ensembles for regularized nonlinear regression](https://mlsanity.com/paper/2002.03375/) (2020)
- [Hidden Markov P'olya trees for high-dimensional distributions](https://mlsanity.com/paper/2011.03121/) (2020)

## Built on by (most cited first, 5 of 5)

- [Probabilistic Circuits for Cumulative Distribution Functions](https://mlsanity.com/paper/2408.04229/) (2024)
- [Two-sample comparison through additive tree models for density ratios](https://mlsanity.com/paper/2508.03059/) (2025)
- [Approximate Bayesian Inference via Bitstring Representations](https://mlsanity.com/paper/2508.13598/) (2025)
- [Generative modeling of density regression through tree flows](https://mlsanity.com/paper/2406.05260/) (2024)
- [A partial likelihood approach to tree-based density modeling and its application in Bayesian inference](https://mlsanity.com/paper/2412.11692/) (2024)

## Abstract (arXiv)

We propose an unsupervised tree boosting algorithm for inferring the underlying sampling distribution of an i.i.d. sample based on fitting additive tree ensembles in a fashion analogous to supervised tree boosting. Integral to the algorithm is a new notion of "addition" on probability distributions that leads to a coherent notion of "residualization", i.e., subtracting a probability distribution from an observation to remove the distributional structure from the sampling distribution of the latter. We show that these notions arise naturally for univariate distributions through cumulative distribution function (CDF) transforms and compositions due to several "group-like" properties of univariate CDFs. While the traditional multivariate CDF does not preserve these properties, a new definition of multivariate CDF can restore these properties, thereby allowing the notions of "addition" and "residualization" to be formulated for multivariate settings as well. This then gives rise to the unsupervised boosting algorithm based on forward-stagewise fitting of an additive tree ensemble, which sequentially reduces the Kullback-Leibler divergence from the truth. The algorithm allows analytic evaluation of the fitted density and outputs a generative model that can be readily sampled from. We enhance the algorithm with scale-dependent shrinkage and a two-stage strategy that separately fits the marginals and the copula. The algorithm then performs competitively to state-of-the-art deep-learning approaches in multivariate density estimation on multiple benchmark data sets.