# Improved Variational Inference in Discrete VAEs using Error Correcting Codes

arXiv: https://arxiv.org/abs/2410.07840 · HTML: https://mlsanity.com/paper/2410.07840/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 18 of its 36 references that are in the map.
- First posted on arXiv 10 Oct 2024.

## Builds on (most cited first, 18 of 18)

- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Squeeze-and-Excitation Networks](https://mlsanity.com/paper/1709.01507/) (2017)
- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms](https://mlsanity.com/paper/1708.07747/) (2017)
- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [Generating Diverse High-Fidelity Images with VQ-VAE-2](https://mlsanity.com/paper/1906.00446/) (2019)
- [The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables](https://mlsanity.com/paper/1611.00712/) (2016)
- [Importance Weighted Autoencoders](https://mlsanity.com/paper/1509.00519/) (2015)
- [VAE with a VampPrior](https://mlsanity.com/paper/1705.07120/) (2017)
- [Variational Lossy Autoencoder](https://mlsanity.com/paper/1611.02731/) (2016)
- [Discrete Variational Autoencoders](https://mlsanity.com/paper/1609.02200/) (2016)
- [Practical Lossless Compression with Latent Variables using Bits Back Coding](https://mlsanity.com/paper/1901.04866/) (2019)
- [Overview of Deep Learning-based CSI Feedback in Massive MIMO Systems](https://mlsanity.com/paper/2206.14383/) (2022)
- [DVAE++: Discrete Variational Autoencoders with Overlapping Transformations](https://mlsanity.com/paper/1802.04920/) (2018)
- [DVAE#: Discrete Variational Autoencoders with Relaxed Boltzmann Priors](https://mlsanity.com/paper/1805.07445/) (2018)
- [Alpha-divergence Variational Inference Meets Importance Weighted Auto-Encoders: Methodology and Asymptotics](https://mlsanity.com/paper/2210.06226/) (2022)
- [Sensing Integrated DFT-Spread OFDM Waveform and Deep Learning-powered Receiver Design for Terahertz Integrated Sensing and Communication Systems](https://mlsanity.com/paper/2109.14918/) (2021)
- [The Deep Generative Decoder: MAP estimation of representations improves modeling of single-cell RNA data](https://mlsanity.com/paper/2110.06672/) (2021)

## Abstract (arXiv)

Despite advances in deep probabilistic models, learning discrete latent representations remains challenging. This work introduces a novel method to improve inference in discrete Variational Autoencoders by reframing the inference problem through a generative perspective. We conceptualize the model as a communication system, and propose to leverage Error-Correcting Codes (ECCs) to introduce redundancy in latent representations, allowing the variational posterior to produce more accurate estimates and reduce the variational gap. We present a proof-of-concept using a Discrete Variational Autoencoder with binary latent variables and low-complexity repetition codes, extending it to a hierarchical structure for disentangling global and local data features. Our approach significantly improves generation quality, data reconstruction, and uncertainty calibration, outperforming the uncoded models even when trained with tighter bounds such as the Importance Weighted Autoencoder objective. We also outline the properties that ECCs should possess to be effectively utilized for improved discrete variational inference.