mlsanityData to 14 Oct 2026

Temperon: Full-Time SAM Quality at a Third Less Wall-Clock

arXiv 2609.17575 ↗First posted 31 Jul 2026cs.LG cs.CVcited by 0

Not on the map. Why?

Abstract (arXiv)

Sharpness-aware minimization (SAM) doubles the cost of every training step, yet its benefit concentrates where training ends. We study where an expensive training mode should be spent and propose Temperon: a plain-SGD explorer for the first 43% of the epoch budget, then one scheduled hand-off that gives the entire final cosine anneal to a SAM-wrapped Muon refiner. On CIFAR-10/100, SVHN and Tiny ImageNet (five seeds, times reported as epochs-to-target times an idle-GPU-calibrated epoch cost), Temperon matches the best full-time-SAM recipe on accuracy everywhere while reaching the hardest common target 35%, 34% and 32% sooner on three of the four, and sits a tier above the published SAM+SGD recipe at level cost. Ablations make the attribution exact: the Muon refiner is worth +0.85pp with everything else fixed; the explorer's shape and its restarts are worth nothing, and we withdraw them as contributions. Re-running the closest rival, late-phase SAM, at matched budget shows the frontier: it is fastest to every mid-level target, but the tier the Muon refiner buys (0.83 on CIFAR-100, 0.97 on CIFAR-10) is reached by no SGD-refined method in any seed, and on Tiny ImageNet, where Muon buys no tier, the rival simply wins -- the measured boundary of the method. The allocation law transfers to GPT-2 pretraining (full-SAM quality at -29% wall-clock) and GLUE fine-tuning (never worse than full-time SAM at a third of its SAM cost). Two constants organize the economics: skipping SAM early buys a fixed credit, and a Muon epoch costs 1.50x a SAM+SGD epoch on all four datasets. Finally, the hand-off cannot be timed from the trajectory: under cosine schedules the accuracy curve is plateau-then-surge, so the information lives in the schedule, making the scheduled switch principled rather than convenient. Code and a pip-installable implementation are released.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 15 of its 18 references that are in the map. First posted on arXiv 31 Jul 2026.

Builds on

15 of 18 references.

References · 15

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  3. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 5,045
  4. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  5. 2016Pointer Sentinel Mixture Modelscited by 3,096
  6. 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,417
  7. 2018Averaging Weights Leads to Wider Optima and Better Generalizationcited by 1,209
  8. 2016Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimizationcited by 699
  9. 2025Muon is Scalable for LLM Trainingcited by 428
  10. 2021ASAM: Adaptive Sharpness-Aware Minimization for Scale-Invariant Learning of Deep Neural Networkscited by 293
  11. 2017Improving Generalization Performance by Switching from Adam to SGDcited by 125
  12. 2021Sharpness-Aware Minimization Improves Language Model Generalizationcited by 108
3 more
  1. 2024Sharpness-Aware Minimization Efficiently Selects Flatter Minima Late in Trainingcited by 24
  2. 2026OptiRoulette Optimizer: A New Stochastic Meta-Optimizer for up to 5.3x Faster Convergencecited by 1
  3. 2026Sharpness-Aware Minimization and Muon: Robustness under the Spectral Normcited by 1

Built on by

Not cited yet.