# On learning higher-order cumulants in diffusion models

arXiv: https://arxiv.org/abs/2410.21212 · HTML: https://mlsanity.com/paper/2410.21212/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Placed in Diffusion foundations and sampling, one of 14 directions in Diffusion models.
- Builds on 38 of its 57 references that are in the map.
- First posted on arXiv 28 Oct 2024.

## Placed in

- Diffusion models / [Diffusion foundations and sampling](https://mlsanity.com/diffusion/diffusion-foundations-and-sampling/)

## Builds on (most cited first, 30 of 38)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Generative Modeling by Estimating Gradients of the Data Distribution](https://mlsanity.com/paper/1907.05600/) (2019)
- [Neural Ordinary Differential Equations](https://mlsanity.com/paper/1806.07366/) (2018)
- [Elucidating the Design Space of Diffusion-Based Generative Models](https://mlsanity.com/paper/2206.00364/) (2022)
- [Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains](https://mlsanity.com/paper/2006.10739/) (2020)
- [Variational Inference with Normalizing Flows](https://mlsanity.com/paper/1505.05770/) (2015)
- [Improved Techniques for Training Score-Based Generative Models](https://mlsanity.com/paper/2006.09011/) (2020)
- [Maximum Likelihood Training of Score-Based Diffusion Models](https://mlsanity.com/paper/2101.09258/) (2021)
- [Deep Neural Networks as Gaussian Processes](https://mlsanity.com/paper/1711.00165/) (2017)
- [Gaussian Process Behaviour in Wide Deep Neural Networks](https://mlsanity.com/paper/1804.11271/) (2018)
- [On the Importance of Noise Scheduling for Diffusion Models](https://mlsanity.com/paper/2301.10972/) (2023)
- [Stochastic Normalizing Flows](https://mlsanity.com/paper/2002.06707/) (2020)
- [Tensor Programs I: Wide Feedforward or Recurrent Neural Networks of Any Architecture are Gaussian Processes](https://mlsanity.com/paper/1910.12478/) (2019)
- [Flow-based generative models for Markov chain Monte Carlo in lattice field theory](https://mlsanity.com/paper/1904.12072/) (2019)
- [Equivariant flow-based sampling for lattice gauge theory](https://mlsanity.com/paper/2003.06413/) (2020)
- [A Phase Transition in Diffusion Models Reveals the Hierarchical Nature of Data](https://mlsanity.com/paper/2402.16991/) (2024)
- [Neural Networks and Quantum Field Theory](https://mlsanity.com/paper/2008.08601/) (2020)
- [Improved Noise Schedule for Diffusion Training](https://mlsanity.com/paper/2407.03297/) (2024)
- [Learning Lattice Quantum Field Theories with Equivariant Continuous Flows](https://mlsanity.com/paper/2207.00283/) (2022)
- [Stochastic normalizing flows as non-equilibrium transformations](https://mlsanity.com/paper/2201.08862/) (2022)
- [Detecting and Mitigating Mode-Collapse for Flow-based Sampling of Lattice Field Theories](https://mlsanity.com/paper/2302.14082/) (2023)
- [Scaling Up Machine Learning For Quantum Field Theory with Equivariant Continuous Flows](https://mlsanity.com/paper/2110.02673/) (2021)
- [Diffusion Models as Stochastic Quantization in Lattice Field Theory](https://mlsanity.com/paper/2309.17082/) (2023)
- [Advances in machine-learning-based sampling motivated by lattice quantum chromodynamics](https://mlsanity.com/paper/2309.01156/) (2023)
- [Neural Network Field Theories: Non-Gaussianity, Actions, and Locality](https://mlsanity.com/paper/2307.03223/) (2023)
- [Quantum field-theoretic machine learning](https://mlsanity.com/paper/2102.09449/) (2021)

## Built on by (most cited first, 10 of 10)

- [Scaling of Stochastic Normalizing Flows in SU(3) lattice gauge theory](https://mlsanity.com/paper/2412.00200/) (2024)
- [Physics-Conditioned Diffusion Models for Lattice Gauge Theory](https://mlsanity.com/paper/2502.05504/) (2025)
- [Scaling flow-based approaches for topology sampling in SU(3) gauge theory](https://mlsanity.com/paper/2510.25704/) (2025)
- [Combining complex Langevin dynamics with score-based and energy-based diffusion models](https://mlsanity.com/paper/2510.01328/) (2025)
- [Diffusion models learn distributions generated by complex Langevin dynamics](https://mlsanity.com/paper/2412.01919/) (2024)
- [The neural networks with tensor weights and emergent fermionic Wick rules in the large-width limit](https://mlsanity.com/paper/2507.05303/) (2025)
- [A scalable flow-based approach to mitigate topological freezing](https://mlsanity.com/paper/2601.20708/) (2026)
- [Accurate Sampling from Diffusion Models](https://mlsanity.com/paper/2609.29902/) (2026)
- [Generative Modeling of Stochastic Dynamics for Long-Time Evolution](https://mlsanity.com/paper/2610.00546/) (2026)
- [Machine Learning Meets High-Energy Nuclear Physics: From Pattern Recognition to Physics-Integrated Discovery](https://mlsanity.com/paper/2610.12293/) (2026)

## Abstract (arXiv)

To analyse how diffusion models learn correlations beyond Gaussian ones, we study the behaviour of higher-order cumulants, or connected n-point functions, under both the forward and backward process. We derive explicit expressions for the moment- and cumulant-generating functionals, in terms of the distribution of the initial data and properties of forward process. It is shown analytically that during the forward process higher-order cumulants are conserved in models without a drift, such as the variance-expanding scheme, and that therefore the endpoint of the forward process maintains nontrivial correlations. We demonstrate that since these correlations are encoded in the score function, higher-order cumulants are learnt in the backward process, also when starting from a normal prior. We confirm our analytical results in an exactly solvable toy model with nonzero cumulants and in scalar lattice field theory.