mlsanityData to 7 Oct 2026

Lossless Compression with Latent Variable Models

arXiv 2104.10544 ↗First posted 21 Apr 2021cs.LG cs.AI cs.ITcited by 4

Not on the map. Why?

Abstract (arXiv)

We develop a simple and elegant method for lossless compression using latent variable models, which we call 'bits back with asymmetric numeral systems' (BB-ANS). The method involves interleaving encode and decode steps, and achieves an optimal rate when compressing batches of data. We demonstrate it firstly on the MNIST test set, showing that state-of-the-art lossless compression is possible using a small variational autoencoder (VAE) model. We then make use of a novel empirical insight, that fully convolutional generative models, trained on small images, are able to generalize to images of arbitrary size, and extend BB-ANS to hierarchical latent variable models, enabling state-of-the-art lossless compression of full-size colour images from the ImageNet dataset. We describe 'Craystack', a modular software framework which we have developed for rapid prototyping of compression using deep generative models.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 34 of its 73 references that are in the map. First posted on arXiv 21 Apr 2021.

Builds on

34 of 73 references.

References · 34

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  3. 2013Auto-Encoding Variational Bayescited by 3,980
  4. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,291
  5. 2016WaveNet: A Generative Model for Raw Audiocited by 1,710
  6. 2018Variational image compression with a scale hyperpriorcited by 910
  7. 2016Pixel Recurrent Neural Networkscited by 814
  8. 2016End-to-end Optimized Image Compressioncited by 673
  9. 2018Joint Autoregressive and Hierarchical Priors for Learned Image Compressioncited by 642
  10. 2016Improving Variational Inference with Inverse Autoregressive Flowcited by 636
  11. 2020NVAE: A Deep Hierarchical Variational Autoencodercited by 635
  12. 2013Generating Sequences With Recurrent Neural Networkscited by 549
22 more
  1. 2017PixelCNN++: Improving the PixelCNN with Discretized Logistic Mixture Likelihood and Other Modificationscited by 415
  2. 2016Ladder Variational Autoencoderscited by 315
  3. 2015MADE: Masked Autoencoder for Distribution Estimationcited by 306
  4. 2017Lossy Image Compression with Compressive Autoencoderscited by 289
  5. 2020Very Deep VAEs Generalize Autoregressive Models and Can Outperform Them on Imagescited by 261
  6. 2015DRAW: A Recurrent Neural Network For Image Generationcited by 190
  7. 2016Composing graphical models with neural networks for structured representations and fast inferencecited by 113
  8. 2019BIVA: A Very Deep Hierarchy of Latent Variables for Generative Modelingcited by 109
  9. 2017Soft Weight-Sharing for Neural Network Compressioncited by 99
  10. 2018Practical Full Resolution Learned Lossless Image Compressioncited by 93
  11. 2019Practical Lossless Compression with Latent Variables using Bits Back Codingcited by 85
  12. 2019Integer Discrete Flows and Lossless Compressioncited by 82
  13. 2018Generating High Fidelity Images with Subscale Pixel Networks and Multidimensional Upscalingcited by 74
  14. 2018Deep Generative Models for Distribution-Preserving Lossy Compressioncited by 59
  15. 2019Bit-Swap: Recursive Bits-Back Coding for Lossless Compression with Hierarchical Latent Variablescited by 51
  16. 2019HiLLoC: Lossless Image Compression with Hierarchical Latent Variable Modelscited by 36
  17. 2020IDF++: Analyzing and Improving Integer Discrete Flows for Lossless Compressioncited by 35
  18. 2019Compression with Flows via Local Bits-Back Codingcited by 34
  19. 2020Locally Masked Convolution for Autoregressive Modelscited by 22
  20. 2021Improving Lossless Compression Rates via Monte Carlo Bits-Back Codingcited by 19
  21. 2021Lossless compression with state space models using bits back codingcited by 6
  22. 2016Lightweight compression with encryption based on Asymmetric Numeral Systemscited by 1

Built on by

Cited by year

Citing papers per year: 2021: 2; 2022: 0; 2023: 1; 2024: 1.

4 papers.

  1. 2021Compressing Multisets with Large Alphabets using Bits-Back Codingcited by 4
  2. 2024Entropy Coding of Unordered Data Structurescited by 3
  3. 2023Random Edge Coding: One-Shot Bits-Back Coding of Large Labeled Graphscited by 2
  4. 2021OSOA: One-Shot Online Adaptation of Deep Generative Models for Lossless Compressioncited by 0