mlsanityData to 7 Oct 2026

Improved Techniques for Training Score-Based Generative Models

arXiv 2006.09011 ↗First posted 16 Jun 2020cs.LG cs.CV stat.MLcited by 1,036

On the map: Diffusion models → Diffusion foundations and sampling

Abstract (arXiv)

Score-based generative models can produce high quality image samples comparable to GANs, without requiring adversarial optimization. However, existing training procedures are limited to images of low resolution (typically below 32x32), and can be unstable under some settings. We provide a new theoretical analysis of learning and sampling from score models in high dimensional spaces, explaining existing failure modes and motivating new solutions that generalize across datasets. To enhance stability, we also propose to maintain an exponential moving average of model weights. With these improvements, we can effortlessly scale score-based generative models to images with unprecedented resolutions ranging from 64x64 to 256x256. Our score-based models can generate high-fidelity samples that rival best-in-class GANs on various image datasets, including CelebA, FFHQ, and multiple LSUN categories.

Key facts. Cited by 1,036 papers in the map (data through 7 Oct 2026). Placed in Diffusion foundations and sampling, one of 14 directions in Diffusion models. Builds on 20 of its 32 references that are in the map. First posted on arXiv 16 Jun 2020.

Builds on

20 of 32 references.

References · 20

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 5,986
  3. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,067
  4. 2014Deep Learning Face Attributes in the Wildcited by 3,679
  5. 2017Progressive Growing of GANs for Improved Quality, Stability, and Variationcited by 2,967
  6. 2017Improved Training of Wasserstein GANscited by 2,223
  7. 2018Spectral Normalization for Generative Adversarial Networkscited by 1,618
  8. 2019Generating Diverse High-Fidelity Images with VQ-VAE-2cited by 1,376
  9. 2015LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loopcited by 1,350
  10. 2015Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)cited by 1,061
  11. 2016Conditional Image Generation with PixelCNN Decoderscited by 859
  12. 2019On the Convergence of Adam and Beyondcited by 834
8 more
  1. 2016RefineNet: Multi-Path Refinement Networks for High-Resolution Semantic Segmentationcited by 463
  2. 2018Assessing Generative Models via Precision and Recallcited by 401
  3. 2019Sliced Score Matching: A Scalable Approach to Density and Score Estimationcited by 381
  4. 2018A Note on the Inception Scorecited by 225
  5. 2017BEGAN: Boundary Equilibrium Generative Adversarial Networkscited by 121
  6. 2019HYPE: A Benchmark for Human eYe Perceptual Evaluation of Generative Modelscited by 59
  7. 2019Neural Empirical Bayescited by 46
  8. 2019Bridging the Gap Between f-GANs and Wasserstein GANscited by 23

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2020: 3; 2021: 69; 2022: 134; 2023: 294; 2024: 268; 2025: 187; 2026: 81.

1,036 papers.

Sort:

By area:

  1. 2020Denoising Diffusion Implicit Modelscited by 9,655
  2. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  3. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  4. 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,736
  5. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,595
  6. 2022Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flowcited by 3,550
  7. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,286
  8. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasetscited by 2,526
  9. 2021SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equationscited by 1,903
  10. 2023Consistency Modelscited by 1,778
  11. 2021Structured Denoising Diffusion Models in Discrete State-Spacescited by 1,665
  12. 2021Image Super-Resolution via Iterative Refinementcited by 1,506
38 more
  1. 2020DiffWave: A Versatile Diffusion Model for Audio Synthesiscited by 1,339
  2. 2021Palette: Image-to-Image Diffusion Modelscited by 1,296
  3. 2021Cascaded Diffusion Models for High Fidelity Image Generationcited by 1,155
  4. 2021Variational Diffusion Modelscited by 1,145
  5. 2022Diffusion Models: A Comprehensive Survey of Methods and Applicationscited by 1,107
  6. 2022Human Motion Diffusion Modelcited by 1,062
  7. 2022Diffusion Models in Vision: A Surveycited by 1,008
  8. 2022eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoiserscited by 921
  9. 2023Improving and generalizing flow-based generative models with minibatch optimal transportcited by 835
  10. 2022Pseudo Numerical Methods for Diffusion Models on Manifoldscited by 695
  11. 2022Point-E: A System for Generating 3D Point Clouds from Complex Promptscited by 694
  12. 2021Maximum Likelihood Training of Score-Based Diffusion Modelscited by 670
  13. 2021Score-based Generative Modeling in Latent Spacecited by 634
  14. 2021ILVR: Conditioning Method for Denoising Diffusion Probabilistic Modelscited by 634
  15. 2022Score Jacobian Chaining: Lifting Pretrained 2D Diffusion Models for 3D Generationcited by 597
  16. 2021Diffusion Schr"odinger Bridge with Applications to Score-Based Generative Modelingcited by 594
  17. 2021CSDI: Conditional Score-based Diffusion Models for Probabilistic Time Series Imputationcited by 577
  18. 2023Erasing Concepts from Diffusion Modelscited by 530
  19. 2021Tackling the Generative Learning Trilemma with Denoising Diffusion GANscited by 515
  20. 2021Solving Inverse Problems in Medical Imaging with Score-Based Generative Modelscited by 514
  21. 2022GeoDiff: a Geometric Diffusion Model for Molecular Conformation Generationcited by 511
  22. 2022All are Worth Words: A ViT Backbone for Diffusion Modelscited by 498
  23. 2021Label-Efficient Semantic Segmentation with Diffusion Modelscited by 495
  24. 2023Consistency Trajectory Models: Learning Probability Flow ODE Trajectory of Diffusioncited by 448
  25. 2021Grad-TTS: A Diffusion Probabilistic Model for Text-to-Speechcited by 445
  26. 2023DPOK: Reinforcement Learning for Fine-tuning Text-to-Image Diffusion Modelscited by 431
  27. 2023Improved Techniques for Training Consistency Modelscited by 430
  28. 2023InstaFlow: One Step is Enough for High-Quality Diffusion-Based Text-to-Image Generationcited by 387
  29. 2022TabDDPM: Modelling Tabular Data with Diffusion Modelscited by 373
  30. 2023Open-Vocabulary Panoptic Segmentation with Text-to-Image Diffusion Modelscited by 371
  31. 2024SV3D: Novel Multi-view Synthesis and 3D Generation from a Single Image using Latent Video Diffusioncited by 364
  32. 2022Analog Bits: Generating Discrete Data using Diffusion Models with Self-Conditioningcited by 362
  33. 2022DiffusionDet: Diffusion Model for Object Detectioncited by 354
  34. 2024Retrieval-Augmented Generation for AI-Generated Content: A Surveycited by 352
  35. 2021Knowledge Distillation in Iterative Generative Models for Improved Sampling Speedcited by 344
  36. 2022Understanding Diffusion Models: A Unified Perspectivecited by 337
  37. 2021Autoregressive Denoising Diffusion Models for Multivariate Probabilistic Time Series Forecastingcited by 332
  38. 2021Score-based diffusion models for accelerated MRIcited by 308

50 most cited of 1,036 All 1,036 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2021Score-Based Generative Classifiers
  2. 2023Score Mismatching for Generative Modeling
  3. 2023Assessing Robustness via Score-Based Adversarial Image Generation
  4. 2024Score Neural Operator: A Generative Model for Learning and Generalizing Across Multiple Probability Distributions
  5. 2024A Good Score Does not Lead to A Good Generative Model
  6. 2023SMaRt: Improving GANs with Score Matching Regularity
  7. 2024Score-based generative models break the curse of dimensionality in learning a family of sub-Gaussian probability distributions
  8. 2026Non-asymptotic Convergence of Stochastic Gradient Descent in Score-based Generative Models