# Principled Weight Initialization for Hypernetworks

arXiv: https://arxiv.org/abs/2312.08399 · HTML: https://mlsanity.com/paper/2312.08399/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 39 references that are in the map.
- First posted on arXiv 13 Dec 2023.

## Builds on (most cited first, 27 of 27)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications](https://mlsanity.com/paper/1704.04861/) (2017)
- [Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification](https://mlsanity.com/paper/1502.01852/) (2015)
- [Group Normalization](https://mlsanity.com/paper/1803.08494/) (2018)
- [HyperNetworks](https://mlsanity.com/paper/1609.09106/) (2016)
- [Striving for Simplicity: The All Convolutional Net](https://mlsanity.com/paper/1412.6806/) (2014)
- [On the Convergence of Adam and Beyond](https://mlsanity.com/paper/1904.09237/) (2019)
- [Exact solutions to the nonlinear dynamics of learning in deep linear neural networks](https://mlsanity.com/paper/1312.6120/) (2013)
- [Continual learning with hypernetworks](https://mlsanity.com/paper/1906.00695/) (2019)
- [The Marginal Value of Adaptive Gradient Methods in Machine Learning](https://mlsanity.com/paper/1705.08292/) (2017)
- [MetaPruning: Meta Learning for Automatic Neural Network Channel Pruning](https://mlsanity.com/paper/1903.10258/) (2019)
- [SMASH: One-Shot Model Architecture Search through HyperNetworks](https://mlsanity.com/paper/1708.05344/) (2017)
- [Fixup Initialization: Residual Learning Without Normalization](https://mlsanity.com/paper/1901.09321/) (2019)
- [Graph HyperNetworks for Neural Architecture Search](https://mlsanity.com/paper/1810.05749/) (2018)
- [Stochastic Hyperparameter Optimization through Hypernetworks](https://mlsanity.com/paper/1802.09419/) (2018)
- [Bayesian Hypernetworks](https://mlsanity.com/paper/1710.04759/) (2017)
- [Hypernetwork functional image representation](https://mlsanity.com/paper/1902.10404/) (2019)
- [HyperGAN: A Generative Model for Diverse, Performant Neural Networks](https://mlsanity.com/paper/1901.11058/) (2019)
- [Hypernetwork Knowledge Graph Embeddings](https://mlsanity.com/paper/1808.07018/) (2018)
- [Implicit Weight Uncertainty in Neural Networks](https://mlsanity.com/paper/1711.01297/) (2017)
- [Blow: a single-scale hyperconditioned flow for non-parallel raw-audio voice conversion](https://mlsanity.com/paper/1906.00794/) (2019)
- [Modular Universal Reparameterization: Deep Multi-task Learning Across Diverse Domains](https://mlsanity.com/paper/1906.00097/) (2019)
- [A Generative Model for Sampling High-Performance and Diverse Weights for Neural Networks](https://mlsanity.com/paper/1905.02898/) (2019)
- [HyperST-Net: Hypernetworks for Spatio-Temporal Forecasting](https://mlsanity.com/paper/1809.10889/) (2018)
- [HyperNetworks with statistical filtering for defending adversarial examples](https://mlsanity.com/paper/1711.01791/) (2017)
- [Predictive Uncertainty Quantification with Compound Density Networks](https://mlsanity.com/paper/1902.01080/) (2019)
- [Meta Networks for Neural Style Transfer](https://mlsanity.com/paper/1709.04111/) (2017)

## Built on by (most cited first, 19 of 19)

- [Pareto Set Learning for Multi-Objective Reinforcement Learning](https://mlsanity.com/paper/2501.06773/) (2025)
- [HyperFusion: A Hypernetwork Approach to Multimodal Integration of Tabular and Medical Imaging Data for Predictive Modeling](https://mlsanity.com/paper/2403.13319/) (2024)
- [Principled Weight Initialisation for Input-Convex Neural Networks](https://mlsanity.com/paper/2312.12474/) (2023)
- [HyperMARL: Adaptive Hypernetworks for Multi-Agent RL](https://mlsanity.com/paper/2412.04233/) (2024)
- [HyperVLA: Efficient Inference in Vision-Language-Action Models via Hypernetworks](https://mlsanity.com/paper/2510.04898/) (2025)
- [Backpropagation-Free Multi-modal On-Device Model Adaptation via Cloud-Device Collaboration](https://mlsanity.com/paper/2406.01601/) (2024)
- [HyperCLIP: Adapting Vision-Language models with Hypernetworks](https://mlsanity.com/paper/2412.16777/) (2024)
- [Effect of Optimizer, Initializer, and Architecture of Hypernetworks on Continual Learning from Demonstration](https://mlsanity.com/paper/2401.00524/) (2023)
- [DECODE: Domain-aware Continual Domain Expansion for Motion Prediction](https://mlsanity.com/paper/2411.17917/) (2024)
- [HyDA: Hypernetworks for Test Time Domain Adaptation in Medical Imaging Analysis](https://mlsanity.com/paper/2503.04979/) (2025)
- [Sample-efficient Integration of New Modalities into Large Language Models](https://mlsanity.com/paper/2509.04606/) (2025)
- [D'OH: Decoder-Only Random Hypernetworks for Implicit Neural Representations](https://mlsanity.com/paper/2403.19163/) (2024)
- [Hypernetwork-Based Adaptive Aggregation for Multimodal Multiple-Instance Learning in Predicting Coronary Calcium Debulking](https://mlsanity.com/paper/2601.21479/) (2026)
- [Learning Families of Fixed-Wing Aircraft Control Policies with Hypernetwork-Conditioned Reinforcement Learning](https://mlsanity.com/paper/2604.03392/) (2026)
- [Efficient Unlearning through Maximizing Relearning Convergence Delay](https://mlsanity.com/paper/2604.09391/) (2026)
- [A Robust In-Context Model for Conservation Laws: Injecting Context into Flux Neural Operators via Recurrent Vision Transformers](https://mlsanity.com/paper/2605.05488/) (2026)
- [Hypernetworks for Dynamic Feature Selection](https://mlsanity.com/paper/2605.12278/) (2026)
- [StableGrad: Backward Scale Control without Batch Normalization](https://mlsanity.com/paper/2605.19856/) (2026)
- [Distilling a Modular Reservoir Through a Genomic Bottleneck](https://mlsanity.com/paper/2606.28380/) (2026)

## Abstract (arXiv)

Hypernetworks are meta neural networks that generate weights for a main neural network in an end-to-end differentiable manner. Despite extensive applications ranging from multi-task learning to Bayesian deep learning, the problem of optimizing hypernetworks has not been studied to date. We observe that classical weight initialization methods like Glorot & Bengio (2010) and He et al. (2015), when applied directly on a hypernet, fail to produce weights for the mainnet in the correct scale. We develop principled techniques for weight initialization in hypernets, and show that they lead to more stable mainnet weights, lower training loss, and faster convergence.