# The Hidden Linear Structure in Score-Based Models and its Application

arXiv: https://arxiv.org/abs/2311.10892 · HTML: https://mlsanity.com/paper/2311.10892/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Placed in Diffusion foundations and sampling, one of 14 directions in Diffusion models.
- Builds on 10 of its 12 references that are in the map.
- First posted on arXiv 17 Nov 2023.

## Placed in

- Diffusion models / [Diffusion foundations and sampling](https://mlsanity.com/diffusion/diffusion-foundations-and-sampling/)

## Builds on (most cited first, 10 of 10)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [Elucidating the Design Space of Diffusion-Based Generative Models](https://mlsanity.com/paper/2206.00364/) (2022)
- [StarGAN v2: Diverse Image Synthesis for Multiple Domains](https://mlsanity.com/paper/1912.01865/) (2019)
- [Pseudo Numerical Methods for Diffusion Models on Manifolds](https://mlsanity.com/paper/2202.09778/) (2022)
- [Generalization in diffusion models arises from geometry-adaptive harmonic representations](https://mlsanity.com/paper/2310.02557/) (2023)
- [Diffusion Models Generate Images Like Painters: an Analytical Theory of Outline First, Details Later](https://mlsanity.com/paper/2303.02490/) (2023)
- [Closed-Form Diffusion Models](https://mlsanity.com/paper/2310.12395/) (2023)

## Built on by (most cited first, 15 of 15)

- [Breaking the Curse of Dimensionality: Diffusion Models Efficiently Learn Low-Dimensional Distributions](https://mlsanity.com/paper/2409.02426/) (2024)
- [Understanding Generalizability of Diffusion Models Requires Rethinking the Hidden Gaussian Structure](https://mlsanity.com/paper/2410.24060/) (2024)
- [The Unreasonable Effectiveness of Gaussian Score Approximation for Diffusion Models and its Applications](https://mlsanity.com/paper/2412.09726/) (2024)
- [An Analytical Theory of Spectral Bias in the Learning Dynamics of Diffusion Models](https://mlsanity.com/paper/2503.03206/) (2025)
- [Locality in Image Diffusion Models Emerges from Data Statistics](https://mlsanity.com/paper/2509.09672/) (2025)
- [Exploring Low-Dimensional Subspaces in Diffusion Models for Controllable Image Editing](https://mlsanity.com/paper/2409.02374/) (2024)
- [A Survey on Diffusion Models for Anomaly Detection](https://mlsanity.com/paper/2501.11430/) (2025)
- [Diffusion Counterfactual Generation with Semantic Abduction](https://mlsanity.com/paper/2506.07883/) (2025)
- [Generalization Dynamics of Linear Diffusion Models](https://mlsanity.com/paper/2505.24769/) (2025)
- [A Random Matrix Theory Perspective on the Consistency of Diffusion Models](https://mlsanity.com/paper/2602.02908/) (2026)
- [Geometry-Aware Discretization Error of Diffusion Models](https://mlsanity.com/paper/2605.08392/) (2026)
- [Generalization, memorization, and overfitting for diffusion models trained in the lazy high-dimensional regime](https://mlsanity.com/paper/2608.23938/) (2026)
- [Toward Theoretical Insights into Diffusion Trajectory Distillation via Operator Merging](https://mlsanity.com/paper/2505.16024/) (2025)
- [Latent Diffusion Inversion Requires Understanding the Latent Space](https://mlsanity.com/paper/2511.20592/) (2025)
- [Diffusion Models, Denoiser Architecture and Creativity](https://mlsanity.com/paper/2605.16415/) (2026)

## Abstract (arXiv)

Score-based models have achieved remarkable results in the generative modeling of many domains. By learning the gradient of smoothed data distribution, they can iteratively generate samples from complex distribution e.g. natural images. However, is there any universal structure in the gradient field that will eventually be learned by any neural network? Here, we aim to find such structures through a normative analysis of the score function. First, we derived the closed-form solution to the scored-based model with a Gaussian score. We claimed that for well-trained diffusion models, the learned score at a high noise scale is well approximated by the linear score of Gaussian. We demonstrated this through empirical validation of pre-trained images diffusion model and theoretical analysis of the score function. This finding enabled us to precisely predict the initial diffusion trajectory using the analytical solution and to accelerate image sampling by 15-30% by skipping the initial phase without sacrificing image quality. Our finding of the linear structure in the score-based model has implications for better model design and data pre-processing.