mlsanityData to 14 Oct 2026

Principled Weight Initialization for Hypernetworks

arXiv 2312.08399 ↗First posted 13 Dec 2023cs.LGcited by 19

Not on the map. Why?

Abstract (arXiv)

Hypernetworks are meta neural networks that generate weights for a main neural network in an end-to-end differentiable manner. Despite extensive applications ranging from multi-task learning to Bayesian deep learning, the problem of optimizing hypernetworks has not been studied to date. We observe that classical weight initialization methods like Glorot & Bengio (2010) and He et al. (2015), when applied directly on a hypernet, fail to produce weights for the mainnet in the correct scale. We develop principled techniques for weight initialization in hypernets, and show that they lead to more stable mainnet weights, lower training loss, and faster convergence.

Key facts. Cited by 19 papers in the map (data through 14 Oct 2026). Builds on 27 of its 39 references that are in the map. First posted on arXiv 13 Dec 2023.

Builds on

27 of 39 references.

References · 27

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2017MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applicationscited by 3,845
  3. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,611
  4. 2018Group Normalizationcited by 1,524
  5. 2016HyperNetworkscited by 1,027
  6. 2014Striving for Simplicity: The All Convolutional Netcited by 990
  7. 2019On the Convergence of Adam and Beyondcited by 841
  8. 2013Exact solutions to the nonlinear dynamics of learning in deep linear neural networkscited by 763
  9. 2019Continual learning with hypernetworkscited by 273
  10. 2017The Marginal Value of Adaptive Gradient Methods in Machine Learningcited by 250
  11. 2019MetaPruning: Meta Learning for Automatic Neural Network Channel Pruningcited by 227
  12. 2017SMASH: One-Shot Model Architecture Search through HyperNetworkscited by 202
15 more
  1. 2019Fixup Initialization: Residual Learning Without Normalizationcited by 198
  2. 2018Graph HyperNetworks for Neural Architecture Searchcited by 132
  3. 2018Stochastic Hyperparameter Optimization through Hypernetworkscited by 73
  4. 2017Bayesian Hypernetworkscited by 66
  5. 2019Hypernetwork functional image representationcited by 65
  6. 2019HyperGAN: A Generative Model for Diverse, Performant Neural Networkscited by 51
  7. 2018Hypernetwork Knowledge Graph Embeddingscited by 38
  8. 2017Implicit Weight Uncertainty in Neural Networkscited by 30
  9. 2019Blow: a single-scale hyperconditioned flow for non-parallel raw-audio voice conversioncited by 29
  10. 2019Modular Universal Reparameterization: Deep Multi-task Learning Across Diverse Domainscited by 13
  11. 2019A Generative Model for Sampling High-Performance and Diverse Weights for Neural Networkscited by 12
  12. 2018HyperST-Net: Hypernetworks for Spatio-Temporal Forecastingcited by 11
  13. 2017HyperNetworks with statistical filtering for defending adversarial examplescited by 9
  14. 2019Predictive Uncertainty Quantification with Compound Density Networkscited by 9
  15. 2017Meta Networks for Neural Style Transfercited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 2; 2024: 6; 2025: 4; 2026: 7.

19 papers.

Sort:

By area:

  1. 2025Pareto Set Learning for Multi-Objective Reinforcement Learningcited by 20
  2. 2024HyperFusion: A Hypernetwork Approach to Multimodal Integration of Tabular and Medical Imaging Data for Predictive Modelingcited by 15
  3. 2023Principled Weight Initialisation for Input-Convex Neural Networkscited by 12
  4. 2024HyperMARL: Adaptive Hypernetworks for Multi-Agent RLcited by 11
  5. 2025HyperVLA: Efficient Inference in Vision-Language-Action Models via Hypernetworkscited by 7
  6. 2024Backpropagation-Free Multi-modal On-Device Model Adaptation via Cloud-Device Collaborationcited by 5
  7. 2024HyperCLIP: Adapting Vision-Language models with Hypernetworkscited by 3
  8. 2023Effect of Optimizer, Initializer, and Architecture of Hypernetworks on Continual Learning from Demonstrationcited by 2
  9. 2024DECODE: Domain-aware Continual Domain Expansion for Motion Predictioncited by 2
  10. 2025HyDA: Hypernetworks for Test Time Domain Adaptation in Medical Imaging Analysiscited by 1
  11. 2025Sample-efficient Integration of New Modalities into Large Language Modelscited by 1
  12. 2024D'OH: Decoder-Only Random Hypernetworks for Implicit Neural Representationscited by 0
7 more
  1. 2026Hypernetwork-Based Adaptive Aggregation for Multimodal Multiple-Instance Learning in Predicting Coronary Calcium Debulkingcited by 0
  2. 2026Learning Families of Fixed-Wing Aircraft Control Policies with Hypernetwork-Conditioned Reinforcement Learningcited by 0
  3. 2026Efficient Unlearning through Maximizing Relearning Convergence Delaycited by 0
  4. 2026A Robust In-Context Model for Conservation Laws: Injecting Context into Flux Neural Operators via Recurrent Vision Transformerscited by 0
  5. 2026Hypernetworks for Dynamic Feature Selectioncited by 0
  6. 2026StableGrad: Backward Scale Control without Batch Normalizationcited by 0
  7. 2026Distilling a Modular Reservoir Through a Genomic Bottleneckcited by 0