# Fast Conditional Network Compression Using Bayesian HyperNetworks

arXiv: https://arxiv.org/abs/2205.06404 · HTML: https://mlsanity.com/paper/2205.06404/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 27 references that are in the map.
- First posted on arXiv 13 May 2022.

## Builds on (most cited first, 24 of 24)

- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification](https://mlsanity.com/paper/1502.01852/) (2015)
- [Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer](https://mlsanity.com/paper/1701.06538/) (2017)
- [Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding](https://mlsanity.com/paper/1510.00149/) (2015)
- [Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation](https://mlsanity.com/paper/1308.3432/) (2013)
- [Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference](https://mlsanity.com/paper/1712.05877/) (2017)
- [Variational Inference: A Review for Statisticians](https://mlsanity.com/paper/1601.00670/) (2016)
- [HyperNetworks](https://mlsanity.com/paper/1609.09106/) (2016)
- [A Closer Look at Memorization in Deep Networks](https://mlsanity.com/paper/1706.05394/) (2017)
- [Learning Sparse Neural Networks through L0 Regularization](https://mlsanity.com/paper/1712.01312/) (2017)
- [Variational Dropout and the Local Reparameterization Trick](https://mlsanity.com/paper/1506.02557/) (2015)
- [HAQ: Hardware-Aware Automated Quantization with Mixed Precision](https://mlsanity.com/paper/1811.08886/) (2018)
- [CondConv: Conditionally Parameterized Convolutions for Efficient Inference](https://mlsanity.com/paper/1904.04971/) (2019)
- [Variational Dropout Sparsifies Deep Neural Networks](https://mlsanity.com/paper/1701.05369/) (2017)
- [MetaPruning: Meta Learning for Automatic Neural Network Channel Pruning](https://mlsanity.com/paper/1903.10258/) (2019)
- [Characterising Bias in Compressed Models](https://mlsanity.com/paper/2010.03058/) (2020)
- [Dynamic Channel Pruning: Feature Boosting and Suppression](https://mlsanity.com/paper/1810.05331/) (2018)
- [Bayesian Compression for Deep Learning](https://mlsanity.com/paper/1705.08665/) (2017)
- [DHP: Differentiable Meta Pruning via HyperNetworks](https://mlsanity.com/paper/2003.13683/) (2020)
- [Fashionable Modelling with Flux](https://mlsanity.com/paper/1811.01457/) (2018)
- [You Look Twice: GaterNet for Dynamic Filter Selection in CNNs](https://mlsanity.com/paper/1811.11205/) (2018)
- [Batch-Shaping for Learning Conditional Channel Gated Networks](https://mlsanity.com/paper/1907.06627/) (2019)
- [Structured Variational Learning of Bayesian Neural Networks with Horseshoe Priors](https://mlsanity.com/paper/1806.05975/) (2018)
- [Mathematical Models of Overparameterized Neural Networks](https://mlsanity.com/paper/2012.13982/) (2020)

## Built on by (most cited first, 5 of 5)

- [A Brief Review of Hypernetworks in Deep Learning](https://mlsanity.com/paper/2306.06955/) (2023)
- [HyP-NeRF: Learning Improved NeRF Priors using a HyperNetwork](https://mlsanity.com/paper/2306.06093/) (2023)
- [INR-V: A Continuous Representation Space for Video-based Generative Tasks](https://mlsanity.com/paper/2210.16579/) (2022)
- [Partial Hypernetworks for Continual Learning](https://mlsanity.com/paper/2306.10724/) (2023)
- [D'OH: Decoder-Only Random Hypernetworks for Implicit Neural Representations](https://mlsanity.com/paper/2403.19163/) (2024)

## Abstract (arXiv)

We introduce a conditional compression problem and propose a fast framework for tackling it. The problem is how to quickly compress a pretrained large neural network into optimal smaller networks given target contexts, e.g. a context involving only a subset of classes or a context where only limited compute resource is available. To solve this, we propose an efficient Bayesian framework to compress a given large network into much smaller size tailored to meet each contextual requirement. We employ a hypernetwork to parameterize the posterior distribution of weights given conditional inputs and minimize a variational objective of this Bayesian neural network. To further reduce the network sizes, we propose a new input-output group sparsity factorization of weights to encourage more sparseness in the generated weights. Our methods can quickly generate compressed networks with significantly smaller sizes than baseline methods.