# Magnitude Invariant Parametrizations Improve Hypernetwork Learning

arXiv: https://arxiv.org/abs/2304.07645 · HTML: https://mlsanity.com/paper/2304.07645/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 40 of its 63 references that are in the map.
- First posted on arXiv 15 Apr 2023.

## Builds on (most cited first, 30 of 40)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Adding Conditional Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.05543/) (2023)
- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021)
- [Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification](https://mlsanity.com/paper/1502.01852/) (2015)
- [Gaussian Error Linear Units (GELUs)](https://mlsanity.com/paper/1606.08415/) (2016)
- [Implicit Neural Representations with Periodic Activation Functions](https://mlsanity.com/paper/2006.09661/) (2020)
- [Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains](https://mlsanity.com/paper/2006.10739/) (2020)
- [Identity Mappings in Deep Residual Networks](https://mlsanity.com/paper/1603.05027/) (2016)
- [Decoupled Weight Decay Regularization](https://mlsanity.com/paper/1711.05101/) (2017)
- [Group Normalization](https://mlsanity.com/paper/1803.08494/) (2018)
- [Instance Normalization: The Missing Ingredient for Fast Stylization](https://mlsanity.com/paper/1607.08022/) (2016)
- [Visualizing the Loss Landscape of Neural Nets](https://mlsanity.com/paper/1712.09913/) (2017)
- [VoxelMorph: A Learning Framework for Deformable Medical Image Registration](https://mlsanity.com/paper/1809.05231/) (2018)
- [Self-Normalizing Neural Networks](https://mlsanity.com/paper/1706.02515/) (2017)
- [Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks](https://mlsanity.com/paper/1602.07868/) (2016)
- [How Does Batch Normalization Help Optimization?](https://mlsanity.com/paper/1805.11604/) (2018)
- [Parameter-efficient Multi-task Fine-tuning for Transformers via Shared Hypernetworks](https://mlsanity.com/paper/2106.04489/) (2021)
- [Continual learning with hypernetworks](https://mlsanity.com/paper/1906.00695/) (2019)
- [HyperStyle: StyleGAN Inversion with HyperNetworks for Real Image Editing](https://mlsanity.com/paper/2111.15666/) (2021)
- [Understanding Batch Normalization](https://mlsanity.com/paper/1806.02375/) (2018)
- [Dissecting Adam: The Sign, Magnitude and Variance of Stochastic Gradients](https://mlsanity.com/paper/1705.07774/) (2017)
- [Batch Renormalization: Towards Reducing Minibatch Dependence in Batch-Normalized Models](https://mlsanity.com/paper/1702.03275/) (2017)
- [Self-Tuning Networks: Bilevel Optimization of Hyperparameters using Structured Best-Response Functions](https://mlsanity.com/paper/1903.03088/) (2019)
- [HyperInverter: Improving StyleGAN Inversion via Hypernetwork](https://mlsanity.com/paper/2112.00719/) (2021)
- [Meta-Learning Symmetries by Reparameterization](https://mlsanity.com/paper/2007.02933/) (2020)
- [Stochastic Hyperparameter Optimization through Hypernetworks](https://mlsanity.com/paper/1802.09419/) (2018)

## Built on by (most cited first, 14 of 14)

- [VB-LoRA: Extreme Parameter Efficient Fine-Tuning with Vector Banks](https://mlsanity.com/paper/2405.15179/) (2024)
- [Pareto Set Learning for Multi-Objective Reinforcement Learning](https://mlsanity.com/paper/2501.06773/) (2025)
- [Foundation Models Secretly Understand Neural Network Weights: Enhancing Hypernetwork Architectures with Foundation Models](https://mlsanity.com/paper/2503.00838/) (2025)
- [HyperSpace: Hypernetworks for spacing-adaptive image segmentation](https://mlsanity.com/paper/2407.03681/) (2024)
- [Spanning the Visual Analogy Space with a Weight Basis of LoRAs](https://mlsanity.com/paper/2602.15727/) (2026)
- [DECODE: Domain-aware Continual Domain Expansion for Motion Prediction](https://mlsanity.com/paper/2411.17917/) (2024)
- [Implicit Neural Representation For Accurate CFD Flow Field Prediction](https://mlsanity.com/paper/2408.06486/) (2024)
- [cIDIR: Conditioned Implicit Neural Representation for Regularized Deformable Image Registration](https://mlsanity.com/paper/2507.12953/) (2025)
- [Hypernetwork-Driven Low-Rank Adaptation Across Attention Heads](https://mlsanity.com/paper/2510.04295/) (2025)
- [CryoHype: Reconstructing a thousand cryo-EM structures with transformer-based hypernetworks](https://mlsanity.com/paper/2512.06332/) (2025)
- [(Almost) Free Modality Stitching of Foundation Models](https://mlsanity.com/paper/2507.10015/) (2025)
- [A Robust In-Context Model for Conservation Laws: Injecting Context into Flux Neural Operators via Recurrent Vision Transformers](https://mlsanity.com/paper/2605.05488/) (2026)
- [Hypernetworks for Dynamic Feature Selection](https://mlsanity.com/paper/2605.12278/) (2026)
- [Atelier: Learning Local Self-Supervised Features for CryoEM Volumes via Hypernetworks](https://mlsanity.com/paper/2609.30569/) (2026)

## Abstract (arXiv)

Hypernetworks, neural networks that predict the parameters of another neural network, are powerful models that have been successfully used in diverse applications from image generation to multi-task learning. Unfortunately, existing hypernetworks are often challenging to train. Training typically converges far more slowly than for non-hypernetwork models, and the rate of convergence can be very sensitive to hyperparameter choices. In this work, we identify a fundamental and previously unidentified problem that contributes to the challenge of training hypernetworks: a magnitude proportionality between the inputs and outputs of the hypernetwork. We demonstrate both analytically and empirically that this can lead to unstable optimization, thereby slowing down convergence, and sometimes even preventing any learning. We present a simple solution to this problem using a revised hypernetwork formulation that we call Magnitude Invariant Parametrizations (MIP). We demonstrate the proposed solution on several hypernetwork tasks, where it consistently stabilizes training and achieves faster convergence. Furthermore, we perform a comprehensive ablation study including choices of activation function, normalization strategies, input dimensionality, and hypernetwork architecture; and find that MIP improves training in all scenarios. We provide easy-to-use code that can turn existing networks into MIP-based hypernetworks.