# Neural Network-Based Score Estimation in Diffusion Models: Optimization and Generalization

arXiv: https://arxiv.org/abs/2401.15604 · HTML: https://mlsanity.com/paper/2401.15604/

## Key facts

- Cited by 42 papers in the map (data through 14 Oct 2026).
- Placed in Diffusion foundations and sampling, one of 14 directions in Diffusion models.
- Builds on 43 of its 59 references that are in the map.
- First posted on arXiv 28 Jan 2024.

## Placed in

- Diffusion models / [Diffusion foundations and sampling](https://mlsanity.com/diffusion/diffusion-foundations-and-sampling/)

## Builds on (most cited first, 30 of 43)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [Flow Matching for Generative Modeling](https://mlsanity.com/paper/2210.02747/) (2022)
- [Generative Modeling by Estimating Gradients of the Data Distribution](https://mlsanity.com/paper/1907.05600/) (2019)
- [Neural Tangent Kernel: Convergence and Generalization in Neural Networks](https://mlsanity.com/paper/1806.07572/) (2018)
- [Efficiently Modeling Long Sequences with Structured State Spaces](https://mlsanity.com/paper/2111.00396/) (2021)
- [Variational Inference with Normalizing Flows](https://mlsanity.com/paper/1505.05770/) (2015)
- [Improved Techniques for Training Score-Based Generative Models](https://mlsanity.com/paper/2006.09011/) (2020)
- [Plug and Play Language Models: A Simple Approach to Controlled Text Generation](https://mlsanity.com/paper/1912.02164/) (2019)
- [Simplified State Space Layers for Sequence Modeling](https://mlsanity.com/paper/2208.04933/) (2022)
- [A Convergence Theory for Deep Learning via Over-Parameterization](https://mlsanity.com/paper/1811.03962/) (2018)
- [Gradient Descent Provably Optimizes Over-parameterized Neural Networks](https://mlsanity.com/paper/1810.02054/) (2018)
- [Score-based Generative Modeling in Latent Space](https://mlsanity.com/paper/2106.05931/) (2021)
- [Diffusion Schr"odinger Bridge with Applications to Score-Based Generative Modeling](https://mlsanity.com/paper/2106.01357/) (2021)
- [On Exact Computation with an Infinitely Wide Neural Net](https://mlsanity.com/paper/1904.11955/) (2019)
- [Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networks](https://mlsanity.com/paper/1901.08584/) (2019)
- [Sampling is as easy as learning the score: theory for diffusion models with minimal data assumptions](https://mlsanity.com/paper/2209.11215/) (2022)
- [Learning and Generalization in Overparameterized Neural Networks, Going Beyond Two Layers](https://mlsanity.com/paper/1811.04918/) (2018)
- [Breaking the Curse of Dimensionality with Convex Neural Networks](https://mlsanity.com/paper/1412.8690/) (2014)
- [Deep learning: a statistical viewpoint](https://mlsanity.com/paper/2103.09177/) (2021)
- [Loss landscapes and optimization in over-parameterized non-linear systems and neural networks](https://mlsanity.com/paper/2003.00307/) (2020)
- [Improved Analysis of Score-based Generative Modeling: User-Friendly Bounds under Minimal Smoothness Assumptions](https://mlsanity.com/paper/2211.01916/) (2022)
- [Convergence of score-based generative modeling for general data distributions](https://mlsanity.com/paper/2209.12381/) (2022)
- [Score Approximation, Estimation and Distribution Recovery of Diffusion Models on Low-Dimensional Data](https://mlsanity.com/paper/2302.07194/) (2023)
- [Diffusion Models are Minimax Optimal Distribution Estimators](https://mlsanity.com/paper/2303.01861/) (2023)
- [Gradient Descent with Early Stopping is Provably Robust to Label Noise for Overparameterized Neural Networks](https://mlsanity.com/paper/1903.11680/) (2019)
- [Neural Policy Gradient Methods: Global Optimality and Rates of Convergence](https://mlsanity.com/paper/1909.01150/) (2019)
- [The probability flow ODE is provably fast](https://mlsanity.com/paper/2305.11798/) (2023)
- [Energy-based Generative Adversarial Network](https://mlsanity.com/paper/1609.03126/) (2016)

## Built on by (most cited first, 30 of 42)

- [An Overview of Diffusion Models: Applications, Guided Generation, Statistical Rates and Optimization](https://mlsanity.com/paper/2404.07771/) (2024)
- [Score-based Diffusion Models via Stochastic Differential Equations -- a Technical Tutorial](https://mlsanity.com/paper/2402.07487/) (2024)
- [Breaking the Curse of Dimensionality: Diffusion Models Efficiently Learn Low-Dimensional Distributions](https://mlsanity.com/paper/2409.02426/) (2024)
- [Low-dimensional adaptation of diffusion models: Convergence in total variation](https://mlsanity.com/paper/2501.12982/) (2025)
- [Stochastic Runge-Kutta Methods: Provable Acceleration of Diffusion Models](https://mlsanity.com/paper/2410.04760/) (2024)
- [Provable Acceleration for Diffusion Models under Minimal Assumptions](https://mlsanity.com/paper/2410.23285/) (2024)
- [Evaluating the design space of diffusion-based generative models](https://mlsanity.com/paper/2406.12839/) (2024)
- [Bigger Isn't Always Memorizing: Early Stopping Overparameterized Diffusion Models](https://mlsanity.com/paper/2505.16959/) (2025)
- [Unraveling the Smoothness Properties of Diffusion Models: A Gaussian Mixture Perspective](https://mlsanity.com/paper/2405.16418/) (2024)
- [Generalization through variance: how noise shapes inductive biases in diffusion models](https://mlsanity.com/paper/2504.12532/) (2025)
- [Constrained Diffusion Models via Dual Training](https://mlsanity.com/paper/2408.15094/) (2024)
- [Denoising Score Matching with Random Features: Insights on Diffusion Models from Precise Learning Curves](https://mlsanity.com/paper/2502.00336/) (2025)
- [Stochastic Control for Fine-tuning Diffusion Models: Optimality, Regularity, and Convergence](https://mlsanity.com/paper/2412.18164/) (2024)
- [On the Interpolation Effect of Score Smoothing in Diffusion Models](https://mlsanity.com/paper/2502.19499/) (2025)
- [Differential Privacy Mechanisms in Neural Tangent Kernel Regression](https://mlsanity.com/paper/2407.13621/) (2024)
- [A solvable model of learning generative diffusion: theory and insights](https://mlsanity.com/paper/2501.03937/) (2025)
- [Shallow diffusion networks provably learn hidden low-dimensional structure](https://mlsanity.com/paper/2410.11275/) (2024)
- [Conditional Diffusion Guidance under Hard Constraint: A Stochastic Analysis Approach](https://mlsanity.com/paper/2602.05533/) (2026)
- [Anomaly Detection and Generation with Diffusion Models: A Survey](https://mlsanity.com/paper/2506.09368/) (2025)
- [Algorithm- and Data-Dependent Generalization Bounds for Diffusion Models](https://mlsanity.com/paper/2506.03849/) (2025)
- [Diffusion Models with Heavy-Tailed Targets: Score Estimation and Sampling Guarantees](https://mlsanity.com/paper/2601.06715/) (2026)
- [Diffusion Generative Models Meet Compressed Sensing, with Applications to Imaging and Finance](https://mlsanity.com/paper/2509.03898/) (2025)
- [Schr"odinger bridge for generative AI: Soft-constrained formulation and convergence analysis](https://mlsanity.com/paper/2510.11829/) (2025)
- [Convergence Dynamics of Over-Parameterized Score Matching for a Single Gaussian](https://mlsanity.com/paper/2511.22069/) (2025)
- [Asymptotic Learning Curves for Diffusion Models with Random Features Score and Manifold Data](https://mlsanity.com/paper/2603.22962/) (2026)
- [Tucker Diffusion Model for High-dimensional Tensor Generation](https://mlsanity.com/paper/2604.00481/) (2026)
- [Dimension-free Score Matching and Time Bootstrapping for Diffusion Models](https://mlsanity.com/paper/2502.10354/) (2025)
- [Multi-Subspace Multi-Modal Modeling for Diffusion Models: Estimation, Convergence and Mixture of Experts](https://mlsanity.com/paper/2601.01475/) (2026)
- [A Quantitative Approximation Framework for Flow Distillation in Diffusion Models](https://mlsanity.com/paper/2606.03820/) (2026)
- [Diffusion Models for Adaptive Sequential Data Generation](https://mlsanity.com/paper/2606.06007/) (2026)

## Abstract (arXiv)

Diffusion models have become a leading paradigm in generative AI, with score estimation via denoising score matching as a central component. While recent theory provides strong statistical guarantees, it typically relies on algorithm-agnostic assumptions and treats empirical risk minimization as if it were solved exactly. In practice, however, score functions are parameterized by highly nonconvex neural networks and trained by gradient descent (GD), and it remains unclear whether such practical procedures admit rigorous guarantees. We take a first step toward this question by developing a mathematical framework for score estimation with GD-trained neural networks. Our analysis addresses both optimization and generalization. We introduce a parametric formulation that reduces denoising score matching to a regression problem with noisy labels. This setting poses several challenges, including unbounded inputs, vector-valued outputs, and an additional time variable, which prevent a direct application of existing techniques. We show that, with a suitable design, the dynamics of GD-trained networks can be approximated by a sequence of localized kernel regression problems. We also show that prolonged training on noisy labels leads to overfitting, and derive an early-stopping rule adapted to unbounded domains. As a consequence, we establish the first minimax-optimal generalization bounds for GD-trained neural networks in diffusion models. Experiments on the Credit Default dataset further show that our theory-guided training framework achieves performance comparable to heavily tuned heuristic methods for generating high-fidelity financial tabular data.