# Locality in Image Diffusion Models Emerges from Data Statistics

arXiv: https://arxiv.org/abs/2509.09672 · HTML: https://mlsanity.com/paper/2509.09672/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- Placed in Diffusion foundations and sampling, one of 14 directions in Diffusion models.
- Builds on 22 of its 37 references that are in the map.
- First posted on arXiv 11 Sep 2025.

## Placed in

- Diffusion models / [Diffusion foundations and sampling](https://mlsanity.com/diffusion/diffusion-foundations-and-sampling/)

## Builds on (most cited first, 22 of 22)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms](https://mlsanity.com/paper/1708.07747/) (2017)
- [Elucidating the Design Space of Diffusion-Based Generative Models](https://mlsanity.com/paper/2206.00364/) (2022)
- [Progressive Growing of GANs for Improved Quality, Stability, and Variation](https://mlsanity.com/paper/1710.10196/) (2017)
- [StarGAN v2: Diverse Image Synthesis for Multiple Domains](https://mlsanity.com/paper/1912.01865/) (2019)
- [Convergence of denoising diffusion models under the manifold hypothesis](https://mlsanity.com/paper/2208.05314/) (2022)
- [Generalization in diffusion models arises from geometry-adaptive harmonic representations](https://mlsanity.com/paper/2310.02557/) (2023)
- [On Memorization in Diffusion Models](https://mlsanity.com/paper/2310.02664/) (2023)
- [An analytic theory of creativity in convolutional diffusion models](https://mlsanity.com/paper/2412.20292/) (2024)
- [Understanding Hallucinations in Diffusion Models through Mode Interpolation](https://mlsanity.com/paper/2406.09358/) (2024)
- [Closed-Form Diffusion Models](https://mlsanity.com/paper/2310.12395/) (2023)
- [Towards a Mechanistic Explanation of Diffusion Model Generalization](https://mlsanity.com/paper/2411.19339/) (2024)
- [Understanding Generalizability of Diffusion Models Requires Rethinking the Hidden Gaussian Structure](https://mlsanity.com/paper/2410.24060/) (2024)
- [The Unreasonable Effectiveness of Gaussian Score Approximation for Diffusion Models and its Applications](https://mlsanity.com/paper/2412.09726/) (2024)
- [On the Generalization of Diffusion Model](https://mlsanity.com/paper/2305.14712/) (2023)
- [Interpreting and Improving Diffusion Models from an Optimization Perspective](https://mlsanity.com/paper/2306.04848/) (2023)
- [Does Generation Require Memorization? Creative Diffusion Models using Ambient Diffusion](https://mlsanity.com/paper/2502.21278/) (2025)
- [The Hidden Linear Structure in Score-Based Models and its Application](https://mlsanity.com/paper/2311.10892/) (2023)
- [Nearest Neighbour Score Estimators for Diffusion Generative Models](https://mlsanity.com/paper/2402.08018/) (2024)

## Built on by (most cited first, 25 of 25)

- [Selective Underfitting in Diffusion Models](https://mlsanity.com/paper/2510.01378/) (2025)
- [Provable Separations between Memorization and Generalization in Diffusion Models](https://mlsanity.com/paper/2511.03202/) (2025)
- [Generalization of Diffusion Models Arises with a Balanced Representation Space](https://mlsanity.com/paper/2512.20963/) (2025)
- [Ambient Diffusion Policy: Imitation Learning from Suboptimal Data in Robotics](https://mlsanity.com/paper/2606.12365/) (2026)
- [Local Mechanisms of Compositional Generalization in Conditional Diffusion](https://mlsanity.com/paper/2509.16447/) (2025)
- [Score-based Membership Inference on Diffusion Models](https://mlsanity.com/paper/2509.25003/) (2025)
- [Two Calm Ends and the Wild Middle: A Geometric Picture of Memorization in Diffusion Models](https://mlsanity.com/paper/2602.17846/) (2026)
- [An exact information theory of generalization phase transitions in Bayesian diffusion models](https://mlsanity.com/paper/2607.08041/) (2026)
- [On the Separability of Information in Diffusion Models](https://mlsanity.com/paper/2509.23937/) (2025)
- [Latent Diffusion Inversion Requires Understanding the Latent Space](https://mlsanity.com/paper/2511.20592/) (2025)
- [Diffusion Models Memorize in Training -- and Generalize in Inference](https://mlsanity.com/paper/2603.13419/) (2026)
- [Concurrence of Symmetry Breaking and Nonlocality Phase Transitions in Diffusion Models](https://mlsanity.com/paper/2605.04830/) (2026)
- [The two clocks and the innovation window: When and how generative models learn rules](https://mlsanity.com/paper/2605.10019/) (2026)
- [The critical slowing down in training diffusion models](https://mlsanity.com/paper/2605.12597/) (2026)
- [Efficient and Training-Free Single-Image Diffusion Models](https://mlsanity.com/paper/2606.04299/) (2026)
- [Where the Score Lives: A Wavelet View of Diffusion](https://mlsanity.com/paper/2606.08309/) (2026)
- [Evaluating the Representation Space of Diffusion Models via Self-Supervised Principles](https://mlsanity.com/paper/2606.09718/) (2026)
- [Sparsely Supervised Diffusion](https://mlsanity.com/paper/2602.02699/) (2026)
- [Fast and Scalable Analytical Diffusion](https://mlsanity.com/paper/2602.16498/) (2026)
- [Generalization and Memorization in Rectified Flow](https://mlsanity.com/paper/2603.13421/) (2026)
- [Diffusion Models, Denoiser Architecture and Creativity](https://mlsanity.com/paper/2605.16415/) (2026)
- [Filtered Posterior Mean Collections: A Unified Framework for Analytical Models of Diffusion Generalization](https://mlsanity.com/paper/2605.24192/) (2026)
- [Information Spreading in Diffusion Models from Effective Field Theory](https://mlsanity.com/paper/2608.14308/) (2026)
- [On the spectral properties of generative denoiser Jacobians](https://mlsanity.com/paper/2609.36210/) (2026)
- [Breakdown of Local Denoising as Semantic Speciation](https://mlsanity.com/paper/2609.38176/) (2026)

## Abstract (arXiv)

Recent work has shown that the generalization ability of image diffusion models arises from the locality properties of the trained neural network. In particular, when denoising a particular pixel, the model relies on a limited neighborhood of the input image around that pixel, which, according to the previous work, is tightly related to the ability of these models to produce novel images. Since locality is central to generalization, it is crucial to understand why diffusion models learn local behavior in the first place, as well as the factors that govern the properties of locality patterns. In this work, we present evidence that the locality in deep diffusion models emerges as a statistical property of the image dataset and is not due to the inductive bias of convolutional neural networks, as suggested in previous work. Specifically, we demonstrate that an optimal parametric linear denoiser exhibits similar locality properties to deep neural denoisers. We show, both theoretically and experimentally, that this locality arises directly from pixel correlations present in the image datasets. Moreover, locality patterns are drastically different on specialized datasets, approximating principal components of the data's covariance. We use these insights to craft an analytical denoiser that better matches scores predicted by a deep diffusion model than prior expert-crafted alternatives. Our key takeaway is that while neural network architectures influence generation quality, their primary role is to capture locality patterns inherent in the data.