mlsanityData to 7 Oct 2026

Optimizers for Diffusion Models: A Controlled Benchmark

arXiv 2609.23055 ↗First posted 19 Sep 2026cs.LGcited by 0

On the map: Diffusion models → Discrete and text diffusion also fits: Diffusion foundations and sampling

Abstract (arXiv)

Discrete diffusion models now match autoregressive language models on several benchmarks, while the question of how best to train them has received far less attention: the optimizer is inherited from one paper to the next and never compared. New optimizers, meanwhile, are validated almost exclusively on autoregressive pretraining, a different objective on a different loss surface. We present a controlled optimizer benchmark across four diffusion formulations, to our knowledge the first for discrete diffusion: seven optimizers (AdamW, Lion, Muon, SOAP, MARS, MARS-M, Schedule-Free) on masked diffusion (text8), uniform diffusion (QM9, and LM1B through the Gaussian duality) and Gaussian diffusion on images (CelebA-64), each on a task with published reference values. Every optimizer receives the same search protocol, and every winner is retrained at the full budget with three seeds. AdamW is a strong default but not always the right choice: it is beaten by a resolved margin on two of the four tasks, and the winner changes with the formulation, so the optimizer deserves the same care as the rest of the training recipe. Notably, methods validated on autoregressive language model pretraining transfer well: Muon, MARS-M and SOAP each beat the tuned AdamW on at least one diffusion formulation. The benchmark, all runs and every figure are reproducible end to end from the released code at https://github.com/armanbolatov/diffusion-baselines.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Discrete and text diffusion, one of 14 directions in Diffusion models. Builds on 71 of its 100 references that are in the map. First posted on arXiv 19 Sep 2026.

Builds on

71 of 100 references.

Closest ancestors · 8

Same direction, newest first

  1. 2025Scaling Behavior of Discrete Diffusion Language Models11 Dec
  2. 2025Dream 7B: Diffusion Large Language Models21 Aug
  3. 2025The Diffusion Duality12 Jun
  4. 2025Large Language Diffusion Models14 Feb
  5. 2024Scaling Diffusion Language Models via Adaptation from Autoregressive Models23 Oct
  6. 2024Simple and Effective Masked Diffusion Language Models11 Jun
  7. 2024Simplified and Generalized Masked Diffusion for Discrete Data6 Jun
  8. 2023Discrete Diffusion Modeling by Estimating the Ratios of the Data Distribution25 Oct

Other references · 63

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  4. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  5. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  6. 2020Denoising Diffusion Implicit Modelscited by 9,655
  7. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  8. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  9. 2022Scalable Diffusion Models with Transformerscited by 5,931
  10. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
  11. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,067
  12. 2014Deep Learning Face Attributes in the Wildcited by 3,679
38 more
  1. 2019Optuna: A Next-generation Hyperparameter Optimization Frameworkcited by 2,455
  2. 2021Structured Denoising Diffusion Models in Discrete State-Spacescited by 1,665
  3. 2021Variational Diffusion Modelscited by 1,145
  4. 2017Mixed Precision Trainingcited by 1,135
  5. 2020Improved Techniques for Training Score-Based Generative Modelscited by 1,036
  6. 2015Optimizing Neural Networks with Kronecker-factored Approximate Curvaturecited by 776
  7. 2024MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategiescited by 694
  8. 2016Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimizationcited by 688
  9. 2021Argmax Flows and Multinomial Diffusion: Learning Categorical Distributionscited by 566
  10. 2021MAUVE: Measuring the Gap Between Neural Text and Human Text using Divergence Frontierscited by 466
  11. 2022A Continuous Time Framework for Discrete Denoising Modelscited by 437
  12. 2023Symbolic Discovery of Optimization Algorithmscited by 423
  13. 2018Shampoo: Preconditioned Stochastic Tensor Optimizationcited by 413
  14. 2019Why gradient clipping accelerates training: A theoretical justification for adaptivitycited by 335
  15. 2013One Billion Word Benchmark for Measuring Progress in Statistical Language Modelingcited by 302
  16. 2019Momentum-Based Variance Reduction in Non-Convex SGDcited by 302
  17. 2015A note on the evaluation of generative modelscited by 291
  18. 2017Are GANs Created Equal? A Large-Scale Studycited by 267
  19. 2021Accounting for Variance in Machine Learning Benchmarkscited by 195
  20. 2018Fr'echet ChemNet Distance: A metric for generative models for molecules in drug discoverycited by 191
  21. 2018Measuring the Effects of Data Parallelism on Neural Network Trainingcited by 190
  22. 2025Training Deep Learning Models with Norm-Constrained LMOscited by 177
  23. 2019Show Your Work: Improved Reporting of Experimental Resultscited by 164
  24. 2024Scaling Laws and Compute-Optimal Training Beyond Fixed Training Durationscited by 160
  25. 2024The Road Less Scheduledcited by 158
  26. 2024Simple Guidance Mechanisms for Discrete Diffusion Modelscited by 148
  27. 2023Tree-Structured Parzen Estimator: Understanding Its Algorithm Components and Their Roles for Better Empirical Performancecited by 143
  28. 2020Descending through a Crowded Valley - Benchmarking Deep Learning Optimizerscited by 118
  29. 2019On Empirical Comparisons of Optimizers for Deep Learningcited by 99
  30. 2025Fantastic Pretraining Optimizers and Where to Find Themcited by 87
  31. 2023Revisiting Gradient Clipping: Stochastic bias and tight convergence guaranteescited by 83
  32. 2022Robustness to Unbounded Smoothness of Generalized SignSGDcited by 80
  33. 2024How Does Critical Batch Size Scale in Pre-training?cited by 68
  34. 2024MARS: Unleashing the Power of Variance Reduction for Training Large Modelscited by 59
  35. 2025Benchmarking Optimizers for Large Language Model Pretrainingcited by 53
  36. 2023Lion Secretly Solves Constrained Optimization: As Lyapunov Predictscited by 50
  37. 2018On the Ineffectiveness of Variance Reduced Optimization for Deep Learningcited by 48
  38. 2024The AdEMAMix Optimizer: Better, Faster, Oldercited by 47

50 most cited of 63 All 63 (JSON)

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Diffusion In Diffusion: Reclaiming Global Coherence in Semi-Autoregressive Diffusion
  2. 2026Scaling Beyond Masked Diffusion Language Models
  3. 2025Diffusion Beats Autoregressive in Data-Constrained Settings
  4. 2025Block Diffusion: Interpolating Between Autoregressive and Diffusion Language Models
  5. 2025TESS 2: A Large-Scale Generalist Diffusion Language Model
  6. 2025Diffusion Language Models are Super Data Learners
  7. 2026Consistent Diffusion Language Models
  8. 2026Discrete Diffusion Models: A Unified Framework from Tokenization to Generation