# Learning Product Codebooks using Vector Quantized Autoencoders for Image Retrieval

arXiv: https://arxiv.org/abs/1807.04629 · HTML: https://mlsanity.com/paper/1807.04629/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 Jul 2018.

## Built on by (most cited first, 13 of 13)

- [Disentanglement via Latent Quantization](https://mlsanity.com/paper/2305.18378/) (2023)
- [VQ-GNN: A Universal Framework to Scale up Graph Neural Networks using Vector Quantization](https://mlsanity.com/paper/2110.14363/) (2021)
- [SemTalk: Holistic Co-speech Motion Generation with Frame-level Semantic Emphasis](https://mlsanity.com/paper/2412.16563/) (2024)
- [Towards Variable and Coordinated Holistic Co-Speech Motion Generation](https://mlsanity.com/paper/2404.00368/) (2024)
- [EchoMask: Speech-Queried Attention-based Mask Modeling for Holistic Co-Speech Motion Generation](https://mlsanity.com/paper/2504.09209/) (2025)
- [GestureHYDRA: Semantic Co-speech Gesture Synthesis via Hybrid Modality Diffusion Transformer and Cascaded-Synchronized Retrieval-Augmented Generation](https://mlsanity.com/paper/2507.22731/) (2025)
- [Combo: Co-speech holistic 3D human motion generation and efficient customizable adaptation in harmony](https://mlsanity.com/paper/2408.09397/) (2024)
- [Decomposing Normal and Abnormal Features of Medical Images into Discrete Latent Codes for Content-Based Image Retrieval](https://mlsanity.com/paper/2103.12328/) (2021)
- [Compression Beyond Pixels: Semantic Compression with Multimodal Foundation Models](https://mlsanity.com/paper/2509.05925/) (2025)
- [Cross-Layer Discrete Concept Discovery for Interpreting Language Models](https://mlsanity.com/paper/2506.20040/) (2025)
- [MAG: Multi-Modal Aligned Autoregressive Co-Speech Gesture Generation without Vector Quantization](https://mlsanity.com/paper/2503.14040/) (2025)
- [Unsupervised Domain-adaptive Hash for Networks](https://mlsanity.com/paper/2108.09136/) (2021)
- [Implicit Feature Decoupling with Depthwise Quantization](https://mlsanity.com/paper/2203.08080/) (2022)

## Abstract (arXiv)

Vector-Quantized Variational Autoencoders (VQ-VAE)[1] provide an unsupervised model for learning discrete representations by combining vector quantization and autoencoders. In this paper, we study the use of VQ-VAE for representation learning for downstream tasks, such as image retrieval. We first describe the VQ-VAE in the context of an information-theoretic framework. We show that the regularization term on the learned representation is determined by the size of the embedded codebook before the training and it affects the generalization ability of the model. As a result, we introduce a hyperparameter to balance the strength of the vector quantizer and the reconstruction error. By tuning the hyperparameter, the embedded bottleneck quantizer is used as a regularizer that forces the output of the encoder to share a constrained coding space such that learned latent features preserve the similarity relations of the data space. In addition, we provide a search range for finding the best hyperparameter. Finally, we incorporate the product quantization into the bottleneck stage of VQ-VAE and propose an end-to-end unsupervised learning model for the image retrieval task. The product quantizer has the advantage of generating large-size codebooks. Fast retrieval can be achieved by using the lookup tables that store the distance between any pair of sub-codewords. State-of-the-art retrieval results are achieved by the learned codebooks.