mlsanityData to 7 Oct 2026

Strong convexity-guided hyper-parameter optimization for flatter losses

arXiv 2402.05025 ↗First posted 7 Feb 2024cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

We propose a novel white-box approach to hyper-parameter optimization. Motivated by recent work establishing a relationship between flat minima and generalization, we first establish a relationship between the strong convexity of the loss and its flatness. Based on this, we seek to find hyper-parameter configurations that improve flatness by minimizing the strong convexity of the loss. By using the structure of the underlying neural network, we derive closed-form equations to approximate the strong convexity parameter, and attempt to find hyper-parameters that minimize it in a randomized fashion. Through experiments on 14 classification datasets, we show that our method achieves strong performance at a fraction of the runtime.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 38 of its 75 references that are in the map. First posted on arXiv 7 Feb 2024.

Builds on

38 of 75 references.

References · 38

  1. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,337
  2. 2017Visualizing the Loss Landscape of Neural Netscited by 1,126
  3. 2013Exact solutions to the nonlinear dynamics of learning in deep linear neural networkscited by 748
  4. 2016Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimizationcited by 688
  5. 2017Exploring Generalization in Deep Learningcited by 595
  6. 2015Train faster, generalize better: Stability of stochastic gradient descentcited by 547
  7. 2017Computing Nonvacuous Generalization Bounds for Deep (Stochastic) Neural Networks with Many More Parameters than Training Datacited by 527
  8. 2019Quantifying the Carbon Emissions of Machine Learningcited by 516
  9. 2019Fantastic Generalization Measures and Where to Find Themcited by 507
  10. 2018BOHB: Robust and Efficient Hyperparameter Optimization at Scalecited by 466
  11. 2017Sharp Minima Can Generalize For Deep Netscited by 462
  12. 2019Scalable Global Optimization via Local Bayesian Optimizationcited by 359
26 more
  1. 2016Entropy-SGD: Biasing Gradient Descent Into Wide Valleyscited by 357
  2. 2018How Does Batch Normalization Help Optimization?cited by 355
  3. 2014In Search of the Real Inductive Bias: On the Role of Implicit Regularization in Deep Learningcited by 339
  4. 2017Nearly-tight VC-dimension and pseudodimension bounds for piecewise linear neural networkscited by 304
  5. 2014Identifying and attacking the saddle point problem in high-dimensional non-convex optimizationcited by 286
  6. 2015Scalable Bayesian Optimization Using Deep Neural Networkscited by 284
  7. 2014The Loss Surfaces of Multilayer Networkscited by 278
  8. 2017Three Factors Influencing Minima in SGDcited by 270
  9. 2018A disciplined approach to neural network hyper-parameters: Part 1 -- learning rate, batch size, momentum, and weight decaycited by 223
  10. 2019Limitations of the Empirical Fisher Approximation for Natural Gradient Descentcited by 213
  11. 2014Predictive Entropy Search for Efficient Global Optimization of Black-box Functionscited by 201
  12. 2021Hyperparameter Optimization: Foundations, Algorithms, Best Practices and Open Challengescited by 143
  13. 2023Neural Architecture Search: Insights from 1000 Paperscited by 136
  14. 2021HPOBench: A Collection of Reproducible Multi-Fidelity Benchmark Problems for HPOcited by 111
  15. 2014Freeze-Thaw Bayesian Optimizationcited by 86
  16. 2020HEBO Pushing The Limits of Sample-Efficient Hyperparameter Optimisationcited by 81
  17. 2021HPO-B: A Large-Scale Reproducible Benchmark for Black-Box HPO based on OpenMLcited by 75
  18. 2021YAHPO Gym -- An Efficient Multi-Objective Multi-Fidelity Benchmark for Hyperparameter Optimizationcited by 61
  19. 2023The Implicit Regularization of Dynamical Stability in Stochastic Gradient Descentcited by 44
  20. 2016Unreasonable Effectiveness of Learning Neural Networks: From Accessible States and Robust Ensembles to Basic Algorithmic Schemescited by 40
  21. 2015Subdominant Dense Clusters Allow for Simple Learning and High Computational Performance in Neural Networks with Discrete Synapsescited by 39
  22. 2023Sharpness Minimization Algorithms Do Not Only Minimize Sharpness To Achieve Better Generalizationcited by 39
  23. 2023PriorBand: Practical Hyperparameter Optimization in the Age of Deep Learningcited by 31
  24. 2018Negative Log Likelihood Ratio Loss for Deep Neural Network Classificationcited by 17
  25. 2018500+ Times Faster Than Deep Learning (A Case Study Exploring Faster Methods for Text Mining StackOverflow)cited by 13
  26. 2019How to "DODGE" Complex Software Analytics?cited by 11

Built on by

Not cited yet.