# Image Compression with Product Quantized Masked Image Modeling

arXiv: https://arxiv.org/abs/2212.07372 · HTML: https://mlsanity.com/paper/2212.07372/

## Key facts

- Cited by 33 papers in the map (data through 14 Oct 2026).
- Builds on 52 of its 87 references that are in the map.
- First posted on arXiv 14 Dec 2022.

## Builds on (most cited first, 30 of 52)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018)
- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [Taming Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2012.09841/) (2020)
- [BEiT: BERT Pre-Training of Image Transformers](https://mlsanity.com/paper/2106.08254/) (2021)
- [Variational Inference with Normalizing Flows](https://mlsanity.com/paper/1505.05770/) (2015)
- [Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation](https://mlsanity.com/paper/1308.3432/) (2013)
- [Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network](https://mlsanity.com/paper/1609.04802/) (2016)
- [WaveNet: A Generative Model for Raw Audio](https://mlsanity.com/paper/1609.03499/) (2016)
- [Generating Diverse High-Fidelity Images with VQ-VAE-2](https://mlsanity.com/paper/1906.00446/) (2019)
- [Demystifying MMD GANs](https://mlsanity.com/paper/1801.01401/) (2018)
- [Image Super-Resolution Using Deep Convolutional Networks](https://mlsanity.com/paper/1501.00092/) (2014)
- [MaskGIT: Masked Generative Image Transformer](https://mlsanity.com/paper/2202.04200/) (2022)
- [Variational image compression with a scale hyperprior](https://mlsanity.com/paper/1802.01436/) (2018)
- [Conditional Image Generation with PixelCNN Decoders](https://mlsanity.com/paper/1606.05328/) (2016)
- [End-to-end Optimized Image Compression](https://mlsanity.com/paper/1611.01704/) (2016)
- [CogView: Mastering Text-to-Image Generation via Transformers](https://mlsanity.com/paper/2105.13290/) (2021)
- [Joint Autoregressive and Hierarchical Priors for Learned Image Compression](https://mlsanity.com/paper/1809.02736/) (2018)
- [Vector-quantized Image Modeling with Improved VQGAN](https://mlsanity.com/paper/2110.04627/) (2021)
- [Learned Image Compression with Discretized Gaussian Mixture Likelihoods and Attention Modules](https://mlsanity.com/paper/2001.01568/) (2020)
- [Make-A-Scene: Scene-Based Text-to-Image Generation with Human Priors](https://mlsanity.com/paper/2203.13131/) (2022)
- [PixelCNN++: Improving the PixelCNN with Discretized Logistic Mixture Likelihood and Other Modifications](https://mlsanity.com/paper/1701.05517/) (2017)
- [Deep Learning for Image Super-resolution: A Survey](https://mlsanity.com/paper/1902.06068/) (2019)
- [High-Fidelity Generative Image Compression](https://mlsanity.com/paper/2006.09965/) (2020)
- [XCiT: Cross-Covariance Image Transformers](https://mlsanity.com/paper/2106.09681/) (2021)

## Built on by (most cited first, 30 of 33)

- [Finite Scalar Quantization: VQ-VAE Made Simple](https://mlsanity.com/paper/2309.15505/) (2023)
- [Image and Video Tokenization with Binary Spherical Quantization](https://mlsanity.com/paper/2406.07548/) (2024)
- [Towards image compression with perfect realism at ultra-low bitrates](https://mlsanity.com/paper/2310.10325/) (2023)
- [Improving Statistical Fidelity for Neural Image Compression with Implicit Local Likelihood Models](https://mlsanity.com/paper/2301.11189/) (2023)
- [High-Fidelity Image Compression with Score-based Generative Models](https://mlsanity.com/paper/2305.18231/) (2023)
- [Residual Quantization with Implicit Neural Codebooks](https://mlsanity.com/paper/2401.14732/) (2024)
- [FoldToken: Learning Protein Language via Vector Quantization and Beyond](https://mlsanity.com/paper/2403.09673/) (2024)
- [Generative Latent Coding for Ultra-Low Bitrate Image Compression](https://mlsanity.com/paper/2512.20194/) (2025)
- [Controlling Rate, Distortion, and Realism: Towards a Single Comprehensive Neural Image Compression Model](https://mlsanity.com/paper/2405.16817/) (2024)
- [M2T: Masking Transformers Twice for Faster Decoding](https://mlsanity.com/paper/2304.07313/) (2023)
- [GIViC: Generative Implicit Video Compression](https://mlsanity.com/paper/2503.19604/) (2025)
- [Approaching Rate-Distortion Limits in Neural Compression with Lattice Transform Coding](https://mlsanity.com/paper/2403.07320/) (2024)
- [Qinco2: Vector Compression and Search with Improved Implicit Neural Codebooks](https://mlsanity.com/paper/2501.03078/) (2025)
- [PQA: Exploring the Potential of Product Quantization in DNN Hardware Acceleration](https://mlsanity.com/paper/2305.18334/) (2023)
- [Neural Image Compression Using Masked Sparse Visual Representation](https://mlsanity.com/paper/2309.11661/) (2023)
- [Differentiable Vector Quantization for Rate-Distortion Optimization of Generative Image Compression](https://mlsanity.com/paper/2604.10546/) (2026)
- [Robust and Unbounded Length Generalization in Autoregressive Transformer-Based Text-to-Speech](https://mlsanity.com/paper/2410.22179/) (2024)
- [MeToken: Uniform Micro-environment Token Boosts Post-Translational Modification Prediction](https://mlsanity.com/paper/2411.01856/) (2024)
- [Next-Frame Decoding for Ultra-Low-Bitrate Image Compression with Video Diffusion Priors](https://mlsanity.com/paper/2603.15129/) (2026)
- [Are Visual Recognition Models Robust to Image Compression?](https://mlsanity.com/paper/2304.04518/) (2023)
- [Exploring Effective Mask Sampling Modeling for Neural Image Compression](https://mlsanity.com/paper/2306.05704/) (2023)
- [Robustly overfitting latents for flexible neural image compression](https://mlsanity.com/paper/2401.17789/) (2024)
- [Deep Generative Modeling Reshapes Compression and Transmission: From Efficiency to Resiliency](https://mlsanity.com/paper/2406.06446/) (2024)
- [Soft Tail-dropping for Adaptive Visual Tokenization](https://mlsanity.com/paper/2601.14246/) (2026)
- [A Noise Constrained Diffusion (NC-Diffusion) Framework for High Fidelity Image Compression](https://mlsanity.com/paper/2604.06568/) (2026)
- [Dual-Latent Collaborative Decoding for Fidelity-Perception Balanced Image Compression](https://mlsanity.com/paper/2605.14391/) (2026)
- [Joint Hierarchical Priors and Adaptive Spatial Resolution for Efficient Neural Image Compression](https://mlsanity.com/paper/2307.02273/) (2023)
- [PerCoV2: Ultra-Low Bit-Rate Perceptual Image Compression via Query-Based 1D Multimodal Image Tokens](https://mlsanity.com/paper/2503.09368/) (2025)
- [Inter2Former: Dynamic Hybrid Attention for Efficient High-Precision Interactive](https://mlsanity.com/paper/2507.09612/) (2025)
- [Product-Quantised Image Representation for High-Quality Image Synthesis](https://mlsanity.com/paper/2510.03191/) (2025)

## Abstract (arXiv)

Recent neural compression methods have been based on the popular hyperprior framework. It relies on Scalar Quantization and offers a very strong compression performance. This contrasts from recent advances in image generation and representation learning, where Vector Quantization is more commonly employed. In this work, we attempt to bring these lines of research closer by revisiting vector quantization for image compression. We build upon the VQ-VAE framework and introduce several modifications. First, we replace the vanilla vector quantizer by a product quantizer. This intermediate solution between vector and scalar quantization allows for a much wider set of rate-distortion points: It implicitly defines high-quality quantizers that would otherwise require intractably large codebooks. Second, inspired by the success of Masked Image Modeling (MIM) in the context of self-supervised learning and generative image models, we propose a novel conditional entropy model which improves entropy coding by modelling the co-dependencies of the quantized latent codes. The resulting PQ-MIM model is surprisingly effective: its compression performance on par with recent hyperprior methods. It also outperforms HiFiC in terms of FID and KID metrics when optimized with perceptual losses (e.g. adversarial). Finally, since PQ-MIM is compatible with image generation frameworks, we show qualitatively that it can operate under a hybrid mode between compression and generation, with no further training or finetuning. As a result, we explore the extreme compression regime where an image is compressed into 200 bytes, i.e., less than a tweet.