# Learning Distributions via Monte-Carlo Marginalization

arXiv: https://arxiv.org/abs/2308.06352 · HTML: https://mlsanity.com/paper/2308.06352/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 34 references that are in the map.
- First posted on arXiv 11 Aug 2023.

## Builds on (most cited first, 18 of 18)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Deep Learning Face Attributes in the Wild](https://mlsanity.com/paper/1411.7766/) (2014)
- [Density estimation using Real NVP](https://mlsanity.com/paper/1605.08803/) (2016)
- [Glow: Generative Flow with Invertible 1x1 Convolutions](https://mlsanity.com/paper/1807.03039/) (2018)
- [Diffusion Models in Vision: A Survey](https://mlsanity.com/paper/2209.04747/) (2022)
- [Masked Autoregressive Flow for Density Estimation](https://mlsanity.com/paper/1705.07057/) (2017)
- [Deep Generative Modelling: A Comparative Review of VAEs, GANs, Normalizing Flows, Energy-Based and Autoregressive Models](https://mlsanity.com/paper/2103.04922/) (2021)
- [Learning Energy-Based Models by Diffusion Recovery Likelihood](https://mlsanity.com/paper/2012.08125/) (2020)
- [Deep Label Distribution Learning with Label Ambiguity](https://mlsanity.com/paper/1611.01731/) (2016)
- [Resampling Base Distributions of Normalizing Flows](https://mlsanity.com/paper/2110.15828/) (2021)
- [Offline Imitation Learning with Suboptimal Demonstrations via Relaxed Distribution Matching](https://mlsanity.com/paper/2303.02569/) (2023)
- [Boosting Few-Shot Text Classification via Distribution Estimation](https://mlsanity.com/paper/2303.16764/) (2023)
- [Policy Search by Target Distribution Learning for Continuous Control](https://mlsanity.com/paper/1905.11041/) (2019)
- [OPT-GAN: A Broad-Spectrum Global Optimizer for Black-box Problems by Learning Distribution](https://mlsanity.com/paper/2102.03888/) (2021)
- [Many-to-One Distribution Learning and K-Nearest Neighbor Smoothing for Thoracic Disease Identification](https://mlsanity.com/paper/2102.13269/) (2021)
- [Nonlinear MCMC for Bayesian Machine Learning](https://mlsanity.com/paper/2202.05621/) (2022)
- [Reward-Biased Maximum Likelihood Estimation for Neural Contextual Bandits](https://mlsanity.com/paper/2203.04192/) (2022)

## Built on by (most cited first, 4 of 4)

- [Affine-Transformation-Invariant Image Classification by Differentiable Arithmetic Distribution Module](https://mlsanity.com/paper/2309.00752/) (2023)
- [Bridging Distribution Learning and Image Clustering in High-dimensional Space](https://mlsanity.com/paper/2308.15667/) (2023)
- [Is Deep Learning Network Necessary for Image Generation?](https://mlsanity.com/paper/2308.13612/) (2023)
- [Deep Clustering via Distribution Learning](https://mlsanity.com/paper/2408.03407/) (2024)

## Abstract (arXiv)

We propose a novel method to learn intractable distributions from their samples. The main idea is to use a parametric distribution model, such as a Gaussian Mixture Model (GMM), to approximate intractable distributions by minimizing the KL-divergence. Based on this idea, there are two challenges that need to be addressed. First, the computational complexity of KL-divergence is unacceptable when the dimensions of distributions increases. The Monte-Carlo Marginalization (MCMarg) is proposed to address this issue. The second challenge is the differentiability of the optimization process, since the target distribution is intractable. We handle this problem by using Kernel Density Estimation (KDE). The proposed approach is a powerful tool to learn complex distributions and the entire process is differentiable. Thus, it can be a better substitute of the variational inference in variational auto-encoders (VAE). One strong evidence of the benefit of our method is that the distributions learned by the proposed approach can generate better images even based on a pre-trained VAE's decoder. Based on this point, we devise a distribution learning auto-encoder which is better than VAE under the same network architecture. Experiments on standard dataset and synthetic data demonstrate the efficiency of the proposed approach.