# Entropy-Coded MS-VQ-VAE with Learned Priors for Ultra-Low Bitrate Video Compression

arXiv: https://arxiv.org/abs/2607.02562 · HTML: https://mlsanity.com/paper/2607.02562/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 28 references that are in the map.
- First posted on arXiv 28 Jun 2026.

## Builds on (most cited first, 19 of 19)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [Group Normalization](https://mlsanity.com/paper/1803.08494/) (2018)
- [Generating Diverse High-Fidelity Images with VQ-VAE-2](https://mlsanity.com/paper/1906.00446/) (2019)
- [Variational image compression with a scale hyperprior](https://mlsanity.com/paper/1802.01436/) (2018)
- [Pixel Recurrent Neural Networks](https://mlsanity.com/paper/1601.06759/) (2016)
- [Joint Autoregressive and Hierarchical Priors for Learned Image Compression](https://mlsanity.com/paper/1809.02736/) (2018)
- [VideoGPT: Video Generation using VQ-VAE and Transformers](https://mlsanity.com/paper/2104.10157/) (2021)
- [Language Model Beats Diffusion -- Tokenizer is Key to Visual Generation](https://mlsanity.com/paper/2310.05737/) (2023)
- [MAGVIT: Masked Generative Video Transformer](https://mlsanity.com/paper/2212.05199/) (2022)
- [High-Fidelity Generative Image Compression](https://mlsanity.com/paper/2006.09965/) (2020)
- [DVC: An End-to-end Deep Video Compression Framework](https://mlsanity.com/paper/1812.00101/) (2018)
- [Deep Contextual Video Compression](https://mlsanity.com/paper/2109.15047/) (2021)
- [PixelSNAIL: An Improved Autoregressive Generative Model](https://mlsanity.com/paper/1712.09763/) (2017)
- [Nonlinear Transform Coding](https://mlsanity.com/paper/2007.03034/) (2020)
- [Generative Latent Coding for Ultra-Low Bitrate Image Compression](https://mlsanity.com/paper/2512.20194/) (2025)
- [TVC: Tokenized Video Compression with Ultra-Low Bit Rate](https://mlsanity.com/paper/2504.16953/) (2025)
- [Hierarchical Vector-Quantized Latents for Perceptual Low-Resolution Video Compression](https://mlsanity.com/paper/2512.24547/) (2025)

## Built on by (most cited first, 1 of 1)

- [Codebook Capacity Governs Perceptual Quality Across Resolutions in Hierarchical Discrete Video Compression](https://mlsanity.com/paper/2607.23366/) (2026)

## Abstract (arXiv)

Learned video codecs based on continuous latent representations struggle to operate reliably below 0.1 bits per pixel~(bpp): without a differentiable rate signal, Lagrangian optimisation cannot effectively trade reconstruction quality for bitrate at extreme compression ratios. We demonstrate that discrete latent representations sidestep this limitation entirely. In a vector-quantized~(VQ) codec, the codebook size~K imposes a hard information ceiling of log2 K bits per symbol; a learned autoregressive prior then exploits the non-uniform distribution of code usage -- which we show follows a power law -- to push actual bitrates well below this ceiling, without any rate-penalty tuning. Building on the MS-VQ-VAE architecture introduced in~citekotthapalli2026msvqvae, we sweep K ∈ 128, 256, 512, 1024 under a uniform training protocol to trace four operating points on the rate-distortion~(RD) curve. We identify and resolve a critical training instability: gradient-based VQ collapses catastrophically at K ≤ 512, whereas EMA-stabilised codebook updates with dead-code restart maintain full utilisation across all configurations. On 500 UCF101 test clips (64!×!64, 32~frames), our models operate at 0.043-0.064~bpp -- 3.3-5× below H.264's practical floor and 5-7.6× below H.265's floor at this resolution. Every MS-VQ-VAE configuration outperforms H.265 CRF,36 on perceptual quality (LPIPS) despite using 5-7.6× fewer bits. At K=1024, the model surpasses H.265 CRF,36 on LPIPS by a margin of 0.072 absolute while using 5.1× fewer bits. Codebook analysis confirms power-law index distributions and 70-85% entropy efficiency, establishing the pipeline as a principled learned entropy coder.