mlsanityData to 7 Oct 2026

Stochastic Hyperparameter Optimization through Hypernetworks

arXiv 1802.09419 ↗First posted 26 Feb 2018cs.LGcited by 72

Not on the map. Why?

Abstract (arXiv)

Machine learning models are often tuned by nesting optimization of model weights inside the optimization of hyperparameters. We give a method to collapse this nested optimization into joint stochastic optimization of weights and hyperparameters. Our process trains a neural network to output approximately optimal weights as a function of hyperparameters. We show that our technique converges to locally optimal weights and hyperparameters for sufficiently large hypernetworks. We compare this method to standard hyperparameter optimization strategies and demonstrate its effectiveness for tuning thousands of hyperparameters.

Key facts. Cited by 72 papers in the map (data through 7 Oct 2026). First posted on arXiv 26 Feb 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 21; 2022: 19; 2023: 16; 2024: 7; 2025: 4; 2026: 5.

72 papers.

Sort:

By area:

  1. 2021Personalized Federated Learning using Hypernetworkscited by 210
  2. 2021Investigating Bi-Level Optimization for Learning and Vision from a Unified Perspective: A Survey and Beyondcited by 151
  3. 2023A Brief Review of Hypernetworks in Deep Learningcited by 102
  4. 2023ATT3D: Amortized Text-to-3D Object Synthesiscited by 87
  5. 2021HyperMorph: Amortized Hyperparameter Learning for Image Registrationcited by 84
  6. 2023Using Large Language Models for Hyperparameter Optimizationcited by 71
  7. 2022Unified Implicit Neural Stylizationcited by 67
  8. 2021A Fully Single Loop Algorithm for Bilevel Optimization without Hessian Inversecited by 53
  9. 2021Unbiased Gradient Estimation in Unrolled Computation Graphs with Persistent Evolution Strategiescited by 52
  10. 2024LATTE3D: Large-scale Amortized Text-To-Enhanced3D Synthesiscited by 39
  11. 2022Learning the Effect of Registration Hyperparameters with HyperMorphcited by 35
  12. 2021Stability and Generalization of Bilevel Programming in Hyperparameter Optimizationcited by 31
38 more
  1. 2022On Implicit Bias in Overparameterized Bilevel Optimizationcited by 30
  2. 2021A General Descent Aggregation Framework for Gradient-based Bi-level Optimizationcited by 28
  3. 2021Meta-Learning to Improve Pre-Trainingcited by 27
  4. 2022Beyond backpropagation: bilevel optimization through implicit differentiation and equilibrium propagationcited by 26
  5. 2022Gradient-based Bi-level Optimization for Deep Learning: A Surveycited by 26
  6. 2022Local Stochastic Bilevel Optimization with Momentum-Based Variance Reductioncited by 20
  7. 2021Efficient differentiable quadratic programming layers: an ADMM approachcited by 19
  8. 2023Principled Weight Initialization for Hypernetworkscited by 19
  9. 2022Multi-Rate VAE: Train Once, Get the Full Rate-Distortion Curvecited by 18
  10. 2025Pareto Set Learning for Multi-Objective Reinforcement Learningcited by 18
  11. 2022Value Function Based Difference-of-Convex Algorithm for Bilevel Hyperparameter Selection Problemscited by 15
  12. 2023Magnitude Invariant Parametrizations Improve Hypernetwork Learningcited by 14
  13. 2022CVAE-H: Conditionalizing Variational Autoencoders via Hypernetworks and Trajectory Forecasting for Autonomous Drivingcited by 13
  14. 2021Complex Momentum for Optimization in Gamescited by 11
  15. 2022Communication-Efficient Robust Federated Learning with Noisy Labelscited by 11
  16. 2023Dynamic Inter-treatment Information Sharing for Individualized Treatment Effects Estimationcited by 11
  17. 2023Amortized Inference for Gaussian Process Hyperparameters of Structured Kernelscited by 10
  18. 2023Combiner and HyperCombiner Networks: Rules to Combine Multimodality MR Images for Prostate Cancer Localisationcited by 10
  19. 2021Regularization-Agnostic Compressed Sensing MRI Reconstruction with Hypernetworkscited by 9
  20. 2021EvoGrad: Efficient Gradient-Based Meta-Learning and Hyperparameter Optimizationcited by 9
  21. 2022OCD: Learning to Overfit with Conditional Diffusion Modelscited by 9
  22. 2025HyperAdaLoRA: Accelerating LoRA Rank Allocation During Training via Hypernetworks without Sacrificing Performancecited by 9
  23. 2021Towards Evaluating the Robustness of Neural Networks Learned by Transductioncited by 8
  24. 2021Lyapunov Exponents for Diversity in Differentiable Gamescited by 8
  25. 2022HMOE: Hypernetwork-based Mixture of Experts for Domain Generalizationcited by 8
  26. 2021Meta Internal Learningcited by 7
  27. 2021Scalable One-Pass Optimisation of High-Dimensional Weight-Update Hyperparameters by Implicit Differentiationcited by 7
  28. 2022Computing Multiple Image Reconstructions with a Single Hypernetworkcited by 7
  29. 2023Low-Variance Gradient Estimation in Unrolled Computation Graphs with ES-Singlecited by 7
  30. 2024JacNet: Learning Functions with Structured Jacobianscited by 7
  31. 2022Task Selection for AutoML System Evaluationcited by 6
  32. 2024Improving Hyperparameter Optimization with Checkpointed Model Weightscited by 5
  33. 2021Online hyperparameter optimization by real-time recurrent learningcited by 4
  34. 2023Learning Physics-Inspired Regularization for Medical Image Registration with Hypernetworkscited by 4
  35. 2026Motion Attribution for Video Generationcited by 4
  36. 2022Learning the Space of Deep Modelscited by 3
  37. 2023Online Algorithmic Recourse by Collective Actioncited by 3
  38. 2026Fine-grained Analysis of Stability and Generalization for Stochastic Bilevel Optimizationcited by 3

50 most cited of 72 All 72 (JSON)