mlsanityData to 14 Oct 2026

Benchmarking Optimizers for MLPs in Tabular Deep Learning

arXiv 2604.15297 ↗First posted 16 Apr 2026cs.LGcited by 5

Not on the map. Why?

Abstract (arXiv)

MLP is a heavily used backbone in modern deep learning (DL) architectures for supervised learning on tabular data, and AdamW is the go-to optimizer used to train tabular DL models. Unlike architecture design, however, the choice of optimizer for tabular DL has not been examined systematically, despite new optimizers showing promise in other domains. To fill this gap, we benchmark 15 optimizers on 17 tabular datasets for training MLP-based models in the standard supervised learning setting under a shared experiment protocol. Our main finding is that the Muon optimizer consistently outperforms AdamW, and thus should be considered a strong and practical choice for practitioners and researchers, if the associated training efficiency overhead is affordable. Additionally, we find exponential moving average of model weights to be a simple yet effective technique that improves AdamW on vanilla MLPs, though its effect is less consistent across model variants.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 35 of its 44 references that are in the map. First posted on arXiv 16 Apr 2026.

Builds on

35 of 44 references.

References · 35

  1. 2019Optuna: A Next-generation Hyperparameter Optimization Frameworkcited by 2,501
  2. 2018Averaging Weights Leads to Wider Optima and Better Generalizationcited by 1,209
  3. 2019On the Variance of the Adaptive Learning Rate and Beyondcited by 772
  4. 2018signSGD: Compressed Optimisation for Non-Convex Problemscited by 571
  5. 2023Symbolic Discovery of Optimization Algorithmscited by 434
  6. 2025Muon is Scalable for LLM Trainingcited by 428
  7. 2018Shampoo: Preconditioned Stochastic Tensor Optimizationcited by 426
  8. 2023Analyzing and Improving the Training Dynamics of Diffusion Modelscited by 402
  9. 2020AdaBelief Optimizer: Adapting Stepsizes by the Belief in Observed Gradientscited by 283
  10. 2022On Embeddings for Numerical Features in Tabular Deep Learningcited by 186
  11. 2025TabArena: A Living Benchmark for Machine Learning on Tabular Datacited by 176
  12. 2022Adan: Adaptive Nesterov Momentum Algorithm for Faster Optimizing Deep Modelscited by 174
23 more
  1. 2024The Road Less Scheduledcited by 163
  2. 2026TabICLv2: A better, faster, scalable, and open tabular foundation modelcited by 144
  3. 2025TabPFN-2.5: Advancing the State of the Art in Tabular Foundation Modelscited by 132
  4. 2024TabM: Advancing Tabular Deep Learning with Parameter-Efficient Ensemblingcited by 113
  5. 2019On Empirical Comparisons of Optimizers for Deep Learningcited by 103
  6. 2024Better by Default: Strong Pre-Tuned MLPs and Boosted Trees on Tabular Datacited by 92
  7. 2023TabR: Tabular Deep Learning Meets Nearest Neighbors in 2023cited by 74
  8. 2024How Does Critical Batch Size Scale in Pre-training?cited by 73
  9. 2024Exponential Moving Average of Weights in Deep Learning: Dynamics and Benefitscited by 71
  10. 2025Benchmarking Optimizers for Large Language Model Pretrainingcited by 68
  11. 2024The AdEMAMix Optimizer: Better, Faster, Oldercited by 54
  12. 2024Cautious Optimizers: Improving Training with One Line of Codecited by 47
  13. 2025Model Merging in Pre-training of Large Language Modelscited by 37
  14. 2024TabReD: Analyzing Pitfalls and Filling the Gaps in Tabular Deep Learning Benchmarkscited by 33
  15. 2025Accelerating Neural Network Training: An Analysis of the AlgoPerf Competitioncited by 32
  16. 2024Revisiting Nearest Neighbor for Tabular Data: A Deep Tabular Baseline Two Decades Latercited by 30
  17. 2024ADOPT: Modified Adam Can Converge with Any β2 with the Optimal Ratecited by 18
  18. 2025Connections between Schedule-Free Optimizers, AdEMAMix, and Accelerated SGD Variantscited by 10
  19. 2025Fantastic (small) Retrievers and How to Train Them: mxbai-edge-colbert-v0 Tech Reportcited by 10
  20. 2025Through the River: Understanding the Benefit of Schedule-Free Methods for Language Model Trainingcited by 9
  21. 2025Optimizing Rank for High-Fidelity Implicit Neural Representationscited by 8
  22. 2024Tabular Data: Is Deep Learning all you need?cited by 6
  23. 2025Unveiling the Role of Data Uncertainty in Tabular Deep Learningcited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 5.

5 papers.

  1. 2026InFlux++: Real and Synthetic Data for Estimating Dynamic Camera Intrinsicscited by 0
  2. 2026Many Optimizers But Only One Training Path: Repeated Resampling for Adaptive Optimizer Selectioncited by 0
  3. 2026Agentic Search Spaces for Tabular Machine Learningcited by 0
  4. 2026Towards Easier-to-Tune Training Algorithms: New Results from AlgoPerfcited by 0
  5. 2026Revisiting Tabular Deep Regression: Objectives and the Limits of Global Early Stoppingcited by 0