mlsanityData to 14 Oct 2026

Scrap Your Schedules with PopDescent

arXiv 2310.14671 ↗First posted 23 Oct 2023cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

In contemporary machine learning workloads, numerous hyper-parameter search algorithms are frequently utilized to efficiently discover high-performing hyper-parameter values, such as learning and regularization rates. As a result, a range of parameter schedules have been designed to leverage the capability of adjusting hyper-parameters during training to enhance loss performance. These schedules, however, introduce new hyper-parameters to be searched and do not account for the current loss values of the models being trained. To address these issues, we propose Population Descent (PopDescent), a progress-aware hyper-parameter tuning technique that employs a memetic, population-based search. By merging evolutionary and local search processes, PopDescent proactively explores hyper-parameter options during training based on their performance. Our trials on standard machine learning vision tasks show that PopDescent converges faster than existing search methods, finding model parameters with test-loss values up to 18% lower, even when considering the use of schedules. Moreover, we highlight the robustness of PopDescent to its initial training parameters, a crucial characteristic for hyper-parameter search techniques.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 11 of its 32 references that are in the map. First posted on arXiv 23 Oct 2023.

Builds on

11 of 32 references.

References · 11

  1. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,349
  2. 2019Optuna: A Next-generation Hyperparameter Optimization Frameworkcited by 2,501
  3. 2014The Loss Surfaces of Multilayer Networkscited by 280
  4. 2015Stochastic modified equations and adaptive stochastic gradient algorithmscited by 142
  5. 2019An Exponential Learning Rate Schedule for Deep Learningcited by 121
  6. 2018An Alternative View: When Does SGD Escape Local Minima?cited by 115
  7. 2019Grid Search, Random Search, Genetic Algorithm: A Big Comparison for NAScited by 83
  8. 2019Weight-space symmetry in deep networks gives rise to permutation saddles, connected by equal-loss valleys across the loss landscapecited by 63
  9. 2019Deep Learning Theory Review: An Optimal Control and Dynamical Systems Perspectivecited by 27
  10. 2018Evolutionary Stochastic Gradient Descent for Optimization of Deep Neural Networkscited by 13
  11. 2023Comparing the Efficacy of Fine-Tuning and Meta-Learning for Few-Shot Policy Imitationcited by 3

Built on by

Not cited yet.