# Selective Underfitting in Diffusion Models

arXiv: https://arxiv.org/abs/2510.01378 · HTML: https://mlsanity.com/paper/2510.01378/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Placed in Diffusion foundations and sampling, one of 14 directions in Diffusion models.
- Builds on 54 of its 69 references that are in the map.
- First posted on arXiv 1 Oct 2025.

## Placed in

- Diffusion models / [Diffusion foundations and sampling](https://mlsanity.com/diffusion/diffusion-foundations-and-sampling/)

## Builds on (most cited first, 30 of 54)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [Scalable Diffusion Models with Transformers](https://mlsanity.com/paper/2212.09748/) (2022)
- [Classifier-Free Diffusion Guidance](https://mlsanity.com/paper/2207.12598/) (2022)
- [Flow Matching for Generative Modeling](https://mlsanity.com/paper/2210.02747/) (2022)
- [Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow](https://mlsanity.com/paper/2209.03003/) (2022)
- [Billion-scale similarity search with GPUs](https://mlsanity.com/paper/1702.08734/) (2017)
- [SiT: Exploring Flow and Diffusion-based Generative Models with Scalable Interpolant Transformers](https://mlsanity.com/paper/2401.08740/) (2024)
- [Extracting Training Data from Diffusion Models](https://mlsanity.com/paper/2301.13188/) (2023)
- [Representation Alignment for Generation: Training Diffusion Transformers Is Easier Than You Think](https://mlsanity.com/paper/2410.06940/) (2024)
- [Benign Overfitting in Linear Regression](https://mlsanity.com/paper/1906.11300/) (2019)
- [Analyzing and Improving the Training Dynamics of Diffusion Models](https://mlsanity.com/paper/2312.02696/) (2023)
- [Reconstruction vs. Generation: Taming Optimization Dilemma in Latent Diffusion Models](https://mlsanity.com/paper/2501.01423/) (2025)
- [MDTv2: Masked Diffusion Transformer is a Strong Image Synthesizer](https://mlsanity.com/paper/2303.14389/) (2023)
- [REPA-E: Unlocking VAE for End-to-End Tuning with Latent Diffusion Transformers](https://mlsanity.com/paper/2504.10483/) (2025)
- [Generalization in diffusion models arises from geometry-adaptive harmonic representations](https://mlsanity.com/paper/2310.02557/) (2023)
- [Fast Training of Diffusion Models with Masked Transformers](https://mlsanity.com/paper/2306.09305/) (2023)
- [On Memorization in Diffusion Models](https://mlsanity.com/paper/2310.02664/) (2023)
- [Score-Based Generative Models Detect Manifolds](https://mlsanity.com/paper/2206.01018/) (2022)
- [An analytic theory of creativity in convolutional diffusion models](https://mlsanity.com/paper/2412.20292/) (2024)
- [EQ-VAE: Equivariance Regularized Latent Space for Improved Generative Image Modeling](https://mlsanity.com/paper/2502.09509/) (2025)
- [Improving the Diffusability of Autoencoders](https://mlsanity.com/paper/2502.14831/) (2025)
- [Benign Overfitting of Constant-Stepsize SGD for Linear Regression](https://mlsanity.com/paper/2103.12692/) (2021)
- [Diffuse and Disperse: Image Generation with Representation Regularization](https://mlsanity.com/paper/2506.09027/) (2025)
- [No Other Representation Component Is Needed: Diffusion Transformers Can Provide Representation Guidance by Themselves](https://mlsanity.com/paper/2505.02831/) (2025)

## Built on by (most cited first, 18 of 18)

- [Diffusion Transformers with Representation Autoencoders](https://mlsanity.com/paper/2510.11690/) (2025)
- [Flow Map Distillation Without Data](https://mlsanity.com/paper/2511.19428/) (2025)
- [Making Reconstruction FID Predictive of Diffusion Generation FID](https://mlsanity.com/paper/2603.05630/) (2026)
- [From Navigation to Refinement: Revealing the Two-Stage Nature of Flow-based Diffusion Models through Oracle Velocity](https://mlsanity.com/paper/2512.02826/) (2025)
- [A Mechanistic Analysis of Sim-and-Real Co-Training in Generative Robot Policies](https://mlsanity.com/paper/2604.13645/) (2026)
- [Biased Generalization in Diffusion Models](https://mlsanity.com/paper/2603.03469/) (2026)
- [Two Calm Ends and the Wild Middle: A Geometric Picture of Memorization in Diffusion Models](https://mlsanity.com/paper/2602.17846/) (2026)
- [MixFlow Training: Alleviating Exposure Bias with Slowed Interpolation Mixture](https://mlsanity.com/paper/2512.19311/) (2025)
- [Memorization in 3D Shape Generation: An Empirical Study](https://mlsanity.com/paper/2512.23628/) (2025)
- [Memorization Control in Diffusion Models from Denoising-centric Perspective](https://mlsanity.com/paper/2601.21348/) (2026)
- [Improving Diffusion Generalization with Weak-to-Strong Segmented Guidance](https://mlsanity.com/paper/2603.20584/) (2026)
- [Efficient and Training-Free Single-Image Diffusion Models](https://mlsanity.com/paper/2606.04299/) (2026)
- [Generative Neural Operators through Diffusion Last Layer](https://mlsanity.com/paper/2602.04139/) (2026)
- [Grokking of Diffusion Models: Case Study on Modular Addition](https://mlsanity.com/paper/2604.17673/) (2026)
- [Mechanisms of Misgeneralization in Physical Sequence Modeling](https://mlsanity.com/paper/2605.20299/) (2026)
- [Filtered Posterior Mean Collections: A Unified Framework for Analytical Models of Diffusion Generalization](https://mlsanity.com/paper/2605.24192/) (2026)
- [Training-Free Imitation Learning with Closed-Form Diffusion Policies](https://mlsanity.com/paper/2606.01238/) (2026)
- [Correction Training for Diffusion Models](https://mlsanity.com/paper/2610.14136/) (2026)

## Abstract (arXiv)

Diffusion models have emerged as the principal paradigm for generative modeling across various domains. During training, they learn the score function, which in turn is used to generate samples at inference. They raise a basic yet unsolved question: which score do they actually learn? In principle, a diffusion model that matches the empirical score in the entire data space would simply reproduce the training data, failing to generate novel samples. Recent work addresses this question by arguing that diffusion models underfit the empirical score due to training-time inductive biases. In this work, we refine this perspective, introducing the notion of selective underfitting: instead of underfitting the score everywhere, better diffusion models more accurately approximate the score in certain regions of input space, while underfitting it in others. We characterize these regions and design empirical interventions to validate our perspective. Our results establish that selective underfitting is essential for understanding diffusion models, yielding new, testable insights into their generalization and generative performance.