# Understanding the Distillation Process from Deep Generative Models to Tractable Probabilistic Circuits

arXiv: https://arxiv.org/abs/2302.08086 · HTML: https://mlsanity.com/paper/2302.08086/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 41 references that are in the map.
- First posted on arXiv 16 Feb 2023.

## Builds on (most cited first, 23 of 23)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [Density estimation using Real NVP](https://mlsanity.com/paper/1605.08803/) (2016)
- [Knowledge Distillation: A Survey](https://mlsanity.com/paper/2006.05525/) (2020)
- [Glow: Generative Flow with Invertible 1x1 Convolutions](https://mlsanity.com/paper/1807.03039/) (2018)
- [Generating Diverse High-Fidelity Images with VQ-VAE-2](https://mlsanity.com/paper/1906.00446/) (2019)
- [Variational Diffusion Models](https://mlsanity.com/paper/2107.00630/) (2021)
- [BIVA: A Very Deep Hierarchy of Latent Variables for Generative Modeling](https://mlsanity.com/paper/1902.02102/) (2019)
- [Einsum Networks: Fast and Scalable Learning of Tractable Probabilistic Circuits](https://mlsanity.com/paper/2004.06231/) (2020)
- [On the Latent Variable Interpretation in Sum-Product Networks](https://mlsanity.com/paper/1601.06180/) (2016)
- [Tractable Regularization of Probabilistic Circuits](https://mlsanity.com/paper/2106.02264/) (2021)
- [Scaling Up Probabilistic Circuits by Latent Variable Distillation](https://mlsanity.com/paper/2210.04398/) (2022)
- [SPFlow: An Easy and Extensible Library for Deep Probabilistic Learning using Sum-Product Networks](https://mlsanity.com/paper/1901.03704/) (2019)
- [Strudel: Learning Structured-Decomposable Probabilistic Circuits](https://mlsanity.com/paper/2007.09331/) (2020)
- [Group Fairness by Probabilistic Modeling with Latent Fair Decisions](https://mlsanity.com/paper/2009.09031/) (2020)
- [Joints in Random Forests](https://mlsanity.com/paper/2006.14937/) (2020)
- [Lossless Compression with Probabilistic Circuits](https://mlsanity.com/paper/2111.11632/) (2021)
- [Continuous Mixtures of Tractable Probabilistic Models](https://mlsanity.com/paper/2209.10584/) (2022)
- [Sparse Probabilistic Circuits via Pruning and Growing](https://mlsanity.com/paper/2211.12551/) (2022)
- [HyperSPNs: Compact and Expressive Probabilistic Circuits](https://mlsanity.com/paper/2112.00914/) (2021)

## Built on by (most cited first, 19 of 19)

- [Scaling Tractable Probabilistic Circuits: A Systems Perspective](https://mlsanity.com/paper/2406.00766/) (2024)
- [Image Inpainting via Tractable Steering of Diffusion Models](https://mlsanity.com/paper/2401.03349/) (2023)
- [What is the Relationship between Tensor Factorizations and Circuits (and How Can We Exploit it)?](https://mlsanity.com/paper/2409.07953/) (2024)
- [On the Relationship Between Monotone and Squared Probabilistic Circuits](https://mlsanity.com/paper/2408.00876/) (2024)
- [Building Expressive and Tractable Probabilistic Generative Models: A Review](https://mlsanity.com/paper/2402.00759/) (2024)
- [Scaling Continuous Latent Variable Models as Probabilistic Integral Circuits](https://mlsanity.com/paper/2406.06494/) (2024)
- [Scaling Probabilistic Circuits via Monarch Matrices](https://mlsanity.com/paper/2506.12383/) (2025)
- [Probabilistic Integral Circuits](https://mlsanity.com/paper/2310.16986/) (2023)
- [Generative Marginalization Models](https://mlsanity.com/paper/2310.12920/) (2023)
- [A Unified Framework for Human-Allied Learning of Probabilistic Circuits](https://mlsanity.com/paper/2405.02413/) (2024)
- [Rethinking Probabilistic Circuit Parameter Learning](https://mlsanity.com/paper/2505.19982/) (2025)
- [Tractable Representation Learning with Probabilistic Circuits](https://mlsanity.com/paper/2507.04385/) (2025)
- [A Tractable Inference Perspective of Offline RL](https://mlsanity.com/paper/2311.00094/) (2023)
- [SFDDM: Single-fold Distillation for Diffusion models](https://mlsanity.com/paper/2405.14961/) (2024)
- [Probabilistic Graph Circuits: Deep Generative Models for Tractable Probabilistic Inference over Graphs](https://mlsanity.com/paper/2503.12162/) (2025)
- [Geometry-Aware Probabilistic Circuits via Voronoi Tessellations](https://mlsanity.com/paper/2603.11946/) (2026)
- [CircuitSynth: Reliable Synthetic Data Generation](https://mlsanity.com/paper/2604.10114/) (2026)
- [Distilling Latent Manifolds: Resolution Extrapolation by Variational Autoencoders](https://mlsanity.com/paper/2603.14536/) (2026)
- [The Expressivity Boundary of Probabilistic Circuits: A Comparison with Large Language Models](https://mlsanity.com/paper/2605.12940/) (2026)

## Abstract (arXiv)

Probabilistic Circuits (PCs) are a general and unified computational framework for tractable probabilistic models that support efficient computation of various inference tasks (e.g., computing marginal probabilities). Towards enabling such reasoning capabilities in complex real-world tasks, Liu et al. (2022) propose to distill knowledge (through latent variable assignments) from less tractable but more expressive deep generative models. However, it is still unclear what factors make this distillation work well. In this paper, we theoretically and empirically discover that the performance of a PC can exceed that of its teacher model. Therefore, instead of performing distillation from the most expressive deep generative model, we study what properties the teacher model and the PC should have in order to achieve good distillation performance. This leads to a generic algorithmic improvement as well as other data-type-specific ones over the existing latent variable distillation pipeline. Empirically, we outperform SoTA TPMs by a large margin on challenging image modeling benchmarks. In particular, on ImageNet32, PCs achieve 4.06 bits-per-dimension, which is only 0.34 behind variational diffusion models (Kingma et al., 2021).