mlsanityData to 7 Oct 2026

Optimization Benchmark for Diffusion Models on Dynamical Systems

arXiv 2510.19376 ↗First posted 22 Oct 2025cs.LG math.OCcited by 5

On the map: Diffusion models → Diffusion foundations and sampling

Abstract (arXiv)

The training of diffusion models is often absent in the evaluation of new optimization techniques. In this work, we benchmark recent optimization algorithms for training a diffusion model for denoising flow trajectories. We observe that Muon and SOAP are highly efficient alternatives to AdamW (18% lower final loss). We also revisit several recent phenomena related to the training of models for text or image applications in the context of diffusion model training. This includes the impact of the learning-rate schedule on the training dynamics, and the performance gap between Adam and SGD.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Placed in Diffusion foundations and sampling, one of 14 directions in Diffusion models. Builds on 26 of its 34 references that are in the map. First posted on arXiv 22 Oct 2025.

Builds on

26 of 34 references.

Closest ancestors · 1

Same direction, newest first

  1. 2020Denoising Diffusion Probabilistic Models19 Jun

Other references · 25

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  2. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  3. 2021Scaling Vision Transformerscited by 993
  4. 2024MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategiescited by 694
  5. 2021Machine learning accelerated computational fluid dynamicscited by 461
  6. 2018Shampoo: Preconditioned Stochastic Tensor Optimizationcited by 413
  7. 2024Scaling Laws and Compute-Optimal Training Beyond Fixed Training Durationscited by 160
  8. 2024The Road Less Scheduledcited by 158
  9. 2023Prodigy: An Expeditiously Adaptive Parameter-Free Learnercited by 129
  10. 2020Descending through a Crowded Valley - Benchmarking Deep Learning Optimizerscited by 118
  11. 2023Noise Is Not the Main Factor Behind the Gap Between SGD and Adam on Transformers, but Sign Descent Might Becited by 114
  12. 2023Score-based Data Assimilationcited by 91
13 more
  1. 2025Fantastic Pretraining Optimizers and Where to Find Themcited by 87
  2. 2024Heavy-Tailed Class Imbalance and Why Adam Outperforms Gradient Descent on Language Modelscited by 78
  3. 2024Modular Duality in Deep Learningcited by 71
  4. 2025Benchmarking Optimizers for Large Language Model Pretrainingcited by 53
  5. 2023Benchmarking Neural Network Training Algorithmscited by 46
  6. 2023Optimal Linear Decay Learning Rate Schedules and Further Refinementscited by 41
  7. 2025Small Batch Size Training for Language Models: When Vanilla SGD Works, and Why Gradient Accumulation Is Wastefulcited by 36
  8. 2025The Surprising Agreement Between Convex Optimization Theory and Learning-Rate Scheduling for Large Model Trainingcited by 31
  9. 2025Accelerating Neural Network Training: An Analysis of the AlgoPerf Competitioncited by 30
  10. 2024Generative Data Assimilation of Sparse Weather Station Observations at Kilometer Scalescited by 27
  11. 2025Appa: Bending Weather Dynamics with Latent Diffusion Models for Global Data Assimilationcited by 17
  12. 2025Is your batch size the problem? Revisiting the Adam-SGD gap in language modelingcited by 14
  13. 2024A Generative Framework for Probabilistic, Spatiotemporally Coherent Downscaling of Climate Simulationcited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 4.

5 papers.

  1. 2025Real-Time Streamable Generative Speech Restoration with Flow Matchingcited by 4
  2. 2026CMuon: Accelerating and Stabilizing Diffusion Transformer Training via Chunked Momentum Orthogonalizationcited by 3
  3. 2026Beyond a Single Explanation of the Adam--SGD Gapcited by 1
  4. 2026Scaling Muon for Diffusion Transformerscited by 1
  5. 2026Optimizers for Diffusion Models: A Controlled Benchmarkcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Diffusion Models: A Mathematical Introduction
  2. 2023The Missing U for Efficient Diffusion Models
  3. 2025Glance: Accelerating Diffusion Models with 1 Sample
  4. 2025Fewer Denoising Steps or Cheaper Per-Step Inference: Towards Compute-Optimal Diffusion Model Deployment
  5. 2025Improved Immiscible Diffusion: Accelerate Diffusion Training by Reducing Its Miscibility
  6. 2023Adaptive Training Meets Progressive Scaling: Elevating Efficiency in Diffusion Models
  7. 2025Accelerate High-Quality Diffusion Models with Inner Loop Feedback
  8. 2025TADA: Improved Diffusion Sampling with Training-free Augmented Dynamics