# Bit-Swap: Recursive Bits-Back Coding for Lossless Compression with Hierarchical Latent Variables

arXiv: https://arxiv.org/abs/1905.06845 · HTML: https://mlsanity.com/paper/1905.06845/

## Key facts

- Cited by 52 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 May 2019.

## Built on by (most cited first, 30 of 52)

- [Variational Diffusion Models](https://mlsanity.com/paper/2107.00630/) (2021)
- [Language Modeling Is Compression](https://mlsanity.com/paper/2309.10668/) (2023)
- [Lossy Image Compression with Quantized Hierarchical VAEs](https://mlsanity.com/paper/2208.13056/) (2022)
- [On the Out-of-distribution Generalization of Probabilistic Image Modelling](https://mlsanity.com/paper/2109.02639/) (2021)
- [QARV: Quantization-Aware ResNet VAE for Lossy Image Compression](https://mlsanity.com/paper/2302.08899/) (2023)
- [iVPF: Numerical Invertible Volume Preserving Flow for Efficient Lossless Compression](https://mlsanity.com/paper/2103.16211/) (2021)
- [iFlow: Numerically Invertible Flows for Efficient Lossless Compression via a Uniform Coder](https://mlsanity.com/paper/2111.00965/) (2021)
- [A Fast Transformer-based General-Purpose Lossless Compressor](https://mlsanity.com/paper/2203.16114/) (2022)
- [Deep Lossy Plus Residual Coding for Lossless and Near-lossless Image Compression](https://mlsanity.com/paper/2209.04847/) (2022)
- [What is the Relationship between Tensor Factorizations and Circuits (and How Can We Exploit it)?](https://mlsanity.com/paper/2409.07953/) (2024)
- [Lossless Compression with Probabilistic Circuits](https://mlsanity.com/paper/2111.11632/) (2021)
- [Improving Lossless Compression Rates via Monte Carlo Bits-Back Coding](https://mlsanity.com/paper/2102.11086/) (2021)
- [Sparse Probabilistic Circuits via Pruning and Growing](https://mlsanity.com/paper/2211.12551/) (2022)
- [Learning Scalable ℓ_∞-constrained Near-lossless Image Compression via Joint Lossy Image and Residual Compression](https://mlsanity.com/paper/2103.17015/) (2021)
- [Towards Empirical Sandwich Bounds on the Rate-Distortion Function](https://mlsanity.com/paper/2111.12166/) (2021)
- [Compression via Pre-trained Transformers: A Study on Byte-Level Multimodal Data](https://mlsanity.com/paper/2410.05078/) (2024)
- [Understanding Entropy Coding With Asymmetric Numeral Systems (ANS): a Statistician's Perspective](https://mlsanity.com/paper/2201.01741/) (2022)
- [Partition and Code: learning how to compress graphs](https://mlsanity.com/paper/2107.01952/) (2021)
- [Generalization Gap in Amortized Inference](https://mlsanity.com/paper/2205.11640/) (2022)
- [IB-DRR: Incremental Learning with Information-Back Discrete Representation Replay](https://mlsanity.com/paper/2104.10588/) (2021)
- [Scaling Continuous Latent Variable Models as Probabilistic Integral Circuits](https://mlsanity.com/paper/2406.06494/) (2024)
- [Probabilistic Integral Circuits](https://mlsanity.com/paper/2310.16986/) (2023)
- [Approximating Human-Like Few-shot Learning with GPT-based Compression](https://mlsanity.com/paper/2308.06942/) (2023)
- [Lossless compression with state space models using bits back coding](https://mlsanity.com/paper/2103.10150/) (2021)
- [GroupedMixer: An Entropy Model with Group-wise Token-Mixers for Learned Image Compression](https://mlsanity.com/paper/2405.01170/) (2024)
- [CALLIC: Content Adaptive Learning for Lossless Image Compression](https://mlsanity.com/paper/2412.17464/) (2024)
- [Split Hierarchical Variational Compression](https://mlsanity.com/paper/2204.02071/) (2022)
- [PILC: Practical Image Lossless Compression with an End-to-end GPU Oriented Neural Framework](https://mlsanity.com/paper/2206.05279/) (2022)
- [Few-Shot Non-Parametric Learning with Deep Latent Variable Model](https://mlsanity.com/paper/2206.11573/) (2022)
- [Lossless Compression with Latent Variable Models](https://mlsanity.com/paper/2104.10544/) (2021)

## Abstract (arXiv)

The bits-back argument suggests that latent variable models can be turned into lossless compression schemes. Translating the bits-back argument into efficient and practical lossless compression schemes for general latent variable models, however, is still an open problem. Bits-Back with Asymmetric Numeral Systems (BB-ANS), recently proposed by Townsend et al. (2019), makes bits-back coding practically feasible for latent variable models with one latent layer, but it is inefficient for hierarchical latent variable models. In this paper we propose Bit-Swap, a new compression scheme that generalizes BB-ANS and achieves strictly better compression rates for hierarchical latent variable models with Markov chain structure. Through experiments we verify that Bit-Swap results in lossless compression rates that are empirically superior to existing techniques. Our implementation is available at https://github.com/fhkingma/bitswap.