mlsanityData to 7 Oct 2026

Soft Truncation: A Universal Training Technique of Score-based Diffusion Model for High Precision Score Estimation

arXiv 2106.05527 ↗First posted 10 Jun 2021cs.LG cs.AI stat.MLcited by 95

On the map: Diffusion models → Diffusion foundations and sampling

Abstract (arXiv)

Recent advances in diffusion models bring state-of-the-art performance on image generation tasks. However, empirical results from previous research in diffusion models imply an inverse correlation between density estimation and sample generation performances. This paper investigates with sufficient empirical evidence that such inverse correlation happens because density estimation is significantly contributed by small diffusion time, whereas sample generation mainly depends on large diffusion time. However, training a score network well across the entire diffusion time is demanding because the loss scale is significantly imbalanced at each diffusion time. For successful training, therefore, we introduce Soft Truncation, a universally applicable training technique for diffusion models, that softens the fixed and static truncation hyperparameter into a random variable. In experiments, Soft Truncation achieves state-of-the-art performance on CIFAR-10, CelebA, CelebA-HQ 256x256, and STL-10 datasets.

Key facts. Cited by 95 papers in the map (data through 7 Oct 2026). Placed in Diffusion foundations and sampling, one of 14 directions in Diffusion models. Builds on 23 of its 46 references that are in the map. First posted on arXiv 10 Jun 2021.

Builds on

23 of 46 references.

Closest ancestors · 7

Same direction, newest first

  1. 2021A Variational Perspective on Diffusion-Based Generative Models and Score Matching5 Jun
  2. 2021Diffusion Models Beat GANs on Image Synthesis11 May
  3. 2021Improved Denoising Diffusion Probabilistic Models18 Feb
  4. 2021Maximum Likelihood Training of Score-Based Diffusion Models22 Jan
  5. 2020Score-Based Generative Modeling through Stochastic Differential Equations26 Nov
  6. 2020Denoising Diffusion Probabilistic Models19 Jun
  7. 2020Improved Techniques for Training Score-Based Generative Models16 Jun

Other references · 16

  1. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 5,986
  2. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
  3. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,067
  4. 2018Neural Ordinary Differential Equationscited by 3,733
  5. 2014Deep Learning Face Attributes in the Wildcited by 3,679
  6. 2017Progressive Growing of GANs for Improved Quality, Stability, and Variationcited by 2,967
  7. 2020Training Generative Adversarial Networks with Limited Datacited by 1,184
  8. 2016Pixel Recurrent Neural Networkscited by 814
  9. 2018Image Transformercited by 770
  10. 2020NVAE: A Deep Hierarchical Variational Autoencodercited by 635
  11. 2015A note on the evaluation of generative modelscited by 291
  12. 2019Flow++: Improving Flow-Based Generative Models with Variational Dequantization and Architecture Designcited by 258
4 more
  1. 2021TransGAN: Two Pure Transformers Can Make One Strong GAN, and That Can Scale Upcited by 162
  2. 2021Solving Schr"odinger Bridges via Maximum Likelihoodcited by 140
  3. 2021On Aliased Resizing and Surprising Subtleties in GAN Evaluationcited by 51
  4. 2021Densely connected normalizing flowscited by 49

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 13; 2023: 33; 2024: 22; 2025: 21; 2026: 6.

95 papers.

Sort:

By area:

  1. 2022Flow Matching for Generative Modelingcited by 5,426
  2. 2022Diffusion Models: A Comprehensive Survey of Methods and Applicationscited by 1,107
  3. 2022Building Normalizing Flows with Stochastic Interpolantscited by 964
  4. 2022All are Worth Words: A ViT Backbone for Diffusion Modelscited by 498
  5. 2023Consistency Trajectory Models: Learning Probability Flow ODE Trajectory of Diffusioncited by 448
  6. 2023MDTv2: Masked Diffusion Transformer is a Strong Image Synthesizercited by 260
  7. 2022Convergence of denoising diffusion models under the manifold hypothesiscited by 237
  8. 2023Score Approximation, Estimation and Distribution Recovery of Diffusion Models on Low-Dimensional Datacited by 171
  9. 2023Solving Linear Inverse Problems Provably via Posterior Sampling with Latent Diffusion Modelscited by 156
  10. 2023Patch Diffusion: Faster and More Data-Efficient Training of Diffusion Modelscited by 131
  11. 2023Decomposed Diffusion Sampler for Accelerating Large-Scale Inverse Problemscited by 129
  12. 2022Score-Based Generative Models Detect Manifoldscited by 121
38 more
  1. 2023Ambient Diffusion: Learning Clean Distributions from Corrupted Datacited by 108
  2. 2023Policy Representation via Diffusion Probability Model for Reinforcement Learningcited by 105
  3. 2022Soft Diffusion: Score Matching for General Corruptionscited by 101
  4. 2022Refining Generative Process with Discriminator Guidance in Score-based Diffusion Modelscited by 98
  5. 2023Reflected Diffusion Modelscited by 81
  6. 2023Score-based Diffusion Models in Function Spacecited by 80
  7. 2022Frido: Feature Pyramid Diffusion for Complex Scene Image Synthesiscited by 75
  8. 2023Diffusion Models Without Attentioncited by 75
  9. 2023Restoration-Degradation Beyond Linear Diffusions: A Non-Asymptotic Analysis For DDIM-Type Samplerscited by 73
  10. 2024Accelerating Diffusion Sampling with Optimized Time Stepscited by 67
  11. 2023Bayesian Flow Networkscited by 62
  12. 2023On the Design Fundamentals of Diffusion Models: A Surveycited by 59
  13. 2023Consistent Diffusion Models: Mitigating Sampling Drift by Learning to be Consistentcited by 58
  14. 2023Improved Techniques for Maximum Likelihood Estimation for Diffusion ODEscited by 54
  15. 2024Deep Generative Models through the Lens of the Manifold Hypothesis: A Survey and New Connectionscited by 47
  16. 2023Patched Denoising Diffusion Models For High-Resolution Image Synthesiscited by 45
  17. 2024Scalable Diffusion Models with State Space Backbonecited by 45
  18. 2023OMS-DPM: Optimizing the Model Schedule for Diffusion Probabilistic Modelscited by 44
  19. 2022Maximum Likelihood Training of Implicit Nonlinear Diffusion Modelscited by 42
  20. 2024Diffusion-RWKV: Scaling RWKV-Like Architectures for Diffusion Modelscited by 41
  21. 2023Continuous-Time Functional Diffusion Processescited by 39
  22. 2023A Theoretical Justification for Image Inpainting using Denoising Diffusion Probabilistic Modelscited by 37
  23. 2023Conditional score-based diffusion models for Bayesian inference in infinite dimensionscited by 37
  24. 2024DiffuseHigh: Training-free Progressive High-Resolution Image Synthesis through Structure Guidancecited by 33
  25. 2025REPA Works Until It Doesn't: Early-Stopped, Holistic Alignment Supercharges Diffusion Trainingcited by 33
  26. 2025Continuously Augmented Discrete Diffusion model for Categorical Generative Modelingcited by 32
  27. 2024Denoising Task Difficulty-based Curriculum for Training Diffusion Modelscited by 26
  28. 2023ScaleLong: Towards More Stable Training of Diffusion Model via Scaling Network Long Skip Connectioncited by 25
  29. 2024Training Unbiased Diffusion Models From Biased Datasetcited by 25
  30. 2025Energy Matching: Unifying Flow Matching and Energy-Based Models for Generative Modelingcited by 25
  31. 2025Consistent Sampling and Simulation: Molecular Dynamics with Energy-Based Diffusion Modelscited by 23
  32. 2023Lipschitz Singularities in Diffusion Modelscited by 21
  33. 2024Deep Diffusion Image Prior for Efficient OOD Adaptation in 3D Inverse Problemscited by 21
  34. 2024An Expectation-Maximization Algorithm for Training Clean Diffusion Models from Corrupted Observationscited by 20
  35. 2023Learning Energy-Based Models by Cooperative Diffusion Recovery Likelihoodcited by 19
  36. 2024Self-Guided Generation of Minority Samples Using Diffusion Modelscited by 19
  37. 2025Training-Free Safe Denoisers for Safe Use of Diffusion Modelscited by 19
  38. 2025Beyond Masked and Unmasked: Discrete Diffusion Models via Partial Maskingcited by 19

50 most cited of 95 All 95 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024From optimal score matching to optimal sampling
  2. 2024A Simple Early Exiting Framework for Accelerated Sampling in Diffusion Models
  3. 2025Bootstrapping Diffusion: Diffusion Model Training Leveraging Partial and Corrupted Data
  4. 2022How Much is Enough? A Study on Diffusion Times in Score-based Generative Models
  5. 2025Score Augmentation for Diffusion Models
  6. 2023Stable Target Field for Reduced Variance Score Estimation in Diffusion Models
  7. 2024Random Walks with Tweedie: A Unified View of Score-Based Diffusion Models
  8. 2023Unmasking Bias in Diffusion Model Training