# Minimal Random Code Learning with Mean-KL Parameterization

arXiv: https://arxiv.org/abs/2307.07816 · HTML: https://mlsanity.com/paper/2307.07816/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 7 of its 15 references that are in the map.
- First posted on arXiv 15 Jul 2023.

## Builds on (most cited first, 7 of 7)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [What is the State of Neural Network Pruning?](https://mlsanity.com/paper/2003.03033/) (2020)
- [Compressing Neural Networks with the Hashing Trick](https://mlsanity.com/paper/1504.04788/) (2015)
- [TensorFlow Distributions](https://mlsanity.com/paper/1711.10604/) (2017)
- [Minimal Random Code Learning: Getting Bits Back from Compressed Model Parameters](https://mlsanity.com/paper/1810.00440/) (2018)
- [Compressing Images by Encoding Their Latent Representations with Relative Entropy Coding](https://mlsanity.com/paper/2010.01185/) (2020)

## Built on by (most cited first, 1 of 1)

- [Training-Free Vector Quantization via Gaussian VAEs](https://mlsanity.com/paper/2512.06609/) (2025)

## Abstract (arXiv)

This paper studies the qualitative behavior and robustness of two variants of Minimal Random Code Learning (MIRACLE) used to compress variational Bayesian neural networks. MIRACLE implements a powerful, conditionally Gaussian variational approximation for the weight posterior Qw and uses relative entropy coding to compress a weight sample from the posterior using a Gaussian coding distribution Pw. To achieve the desired compression rate, DKL[Qw Vert Pw] must be constrained, which requires a computationally expensive annealing procedure under the conventional mean-variance (Mean-Var) parameterization for Qw. Instead, we parameterize Qw by its mean and KL divergence from Pw to constrain the compression cost to the desired value by construction. We demonstrate that variational training with Mean-KL parameterization converges twice as fast and maintains predictive performance after compression. Furthermore, we show that Mean-KL leads to more meaningful variational distributions with heavier tails and compressed weight samples which are more robust to pruning.