mlsanityData to 14 Oct 2026

Improving Hyperparameter Optimization with Checkpointed Model Weights

arXiv 2406.18630 ↗First posted 26 Jun 2024cs.LG cs.AI stat.MLcited by 5

Not on the map. Why?

Abstract (arXiv)

When training deep learning models, the performance depends largely on the selected hyperparameters. However, hyperparameter optimization (HPO) is often one of the most expensive parts of model design. Classical HPO methods treat this as a black-box optimization problem. However, gray-box HPO methods, which incorporate more information about the setup, have emerged as a promising direction for more efficient optimization. For example, using intermediate loss evaluations to terminate bad selections. In this work, we propose an HPO method for neural networks using logged checkpoints of the trained weights to guide future hyperparameter selections. Our method, Forecasting Model Search (FMS), embeds weights into a Gaussian process deep kernel surrogate model, using a permutation-invariant graph metanetwork to be data-efficient with the logged network weights. To facilitate reproducibility and further research, we open-source our code at https://github.com/NVlabs/forecasting-model-search.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 65 of its 108 references that are in the map. First posted on arXiv 26 Jun 2024.

Builds on

65 of 108 references.

References · 65

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  5. 2016Semi-Supervised Classification with Graph Convolutional Networkscited by 11,361
  6. 2019Fast Graph Representation Learning with PyTorch Geometriccited by 2,434
  7. 2018DARTS: Differentiable Architecture Searchcited by 1,798
  8. 2016Neural Architecture Search with Reinforcement Learningcited by 1,499
  9. 2017Deep Setscited by 1,421
  10. 2018Relational inductive biases, deep learning, and graph networkscited by 1,360
  11. 2020Meta-Learning in Neural Networks: A Surveycited by 950
  12. 2016Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimizationcited by 699
38 more
  1. 2018Taskonomy: Disentangling Task Transfer Learningcited by 634
  2. 2018GPyTorch: Blackbox Matrix-Matrix Gaussian Process Inference with GPU Accelerationcited by 609
  3. 2018Do Better ImageNet Models Transfer Better?cited by 565
  4. 2018BOHB: Robust and Efficient Hyperparameter Optimization at Scalecited by 470
  5. 2015Deep Kernel Learningcited by 445
  6. 2019AutoML: A Survey of the State-of-the-Artcited by 390
  7. 2017Population Based Training of Neural Networkscited by 367
  8. 2015Gradient-based Hyperparameter Optimization through Reversible Learningcited by 360
  9. 2023Data Selection for Language Models via Importance Resamplingcited by 329
  10. 2019Optimizing Millions of Hyperparameters by Implicit Differentiationcited by 304
  11. 2017Forward and Reverse Gradient-Based Hyperparameter Optimizationcited by 234
  12. 2015Non-stochastic Best Arm Identification and Hyperparameter Optimizationcited by 221
  13. 2018Implicit Self-Regularization in Deep Neural Networks: Evidence from Random Matrix Theory and Implications for Learningcited by 215
  14. 2016Deep Information Propagationcited by 195
  15. 2020LEEP: A New Measure to Evaluate Transferability of Learned Representationscited by 188
  16. 2019BANANAS: Bayesian Optimization with Neural Architectures for Neural Architecture Searchcited by 179
  17. 2021LogME: Practical Assessment of Pre-trained Models for Transfer Learningcited by 172
  18. 2021Hyperparameter Optimization: Foundations, Algorithms, Best Practices and Open Challengescited by 145
  19. 2019Transferability and Hardness of Supervised Classification Taskscited by 127
  20. 2017Multi-fidelity Bayesian Optimisation with Continuous Approximationscited by 121
  21. 2020Hyper-Parameter Optimization: A Review of Algorithms and Applicationscited by 114
  22. 2018Predicting the Generalization Gap in Deep Networks with Margin Distributionscited by 113
  23. 2020Predicting Neural Network Accuracy from Weightscited by 110
  24. 2023Equivariant Architectures for Learning in Deep Weight Spacescited by 107
  25. 2019Self-Tuning Networks: Bilevel Optimization of Hyperparameters using Structured Best-Response Functionscited by 105
  26. 2021How Powerful are Performance Predictors in Neural Architecture Search?cited by 90
  27. 2023ATT3D: Amortized Text-to-3D Object Synthesiscited by 87
  28. 2022Learning to Learn with Generative Models of Neural Network Checkpointscited by 86
  29. 2020PACOH: Bayes-Optimal Meta-Learning with PAC-Guaranteescited by 85
  30. 2016Multi-Information Source Optimizationcited by 84
  31. 2018Stochastic Hyperparameter Optimization through Hypernetworkscited by 73
  32. 2021Few-Shot Bayesian Optimization with Deep Kernel Surrogatescited by 71
  33. 2023Using Large Language Models for Hyperparameter Optimizationcited by 71
  34. 2019Meta-Learning Acquisition Functions for Transfer Learning in Bayesian Optimizationcited by 70
  35. 2020Classifying the classifier: dissecting the weight space of neural networkscited by 65
  36. 2022Supervised Training of Conditional Monge Mapscited by 61
  37. 2020Multi-Fidelity Bayesian Optimization via Deep Neural Networkscited by 46
  38. 2022Model Zoos: A Dataset of Diverse Populations of Neural Network Modelscited by 39

50 most cited of 65 All 65 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 2; 2026: 2.

5 papers.

  1. 2024JacNet: Learning Functions with Structured Jacobianscited by 7
  2. 2025Score Distillation Sampling for Audio: Source Separation, Synthesis, and Beyondcited by 6
  3. 2026Motion Attribution for Video Generationcited by 5
  4. 2025Iterated Population Based Training with Task-Agnostic Restartscited by 0
  5. 2026Importance-Aware Scheduling for High-Dimensional Hyperparameter Optimizationcited by 0