# LG-VQ: Language-Guided Codebook Learning

arXiv: https://arxiv.org/abs/2405.14206 · HTML: https://mlsanity.com/paper/2405.14206/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 43 references that are in the map.
- First posted on arXiv 23 May 2024.

## Builds on (most cited first, 20 of 20)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision](https://mlsanity.com/paper/2102.05918/) (2021)
- [Align before Fuse: Vision and Language Representation Learning with Momentum Distillation](https://mlsanity.com/paper/2107.07651/) (2021)
- [LXMERT: Learning Cross-Modality Encoder Representations from Transformers](https://mlsanity.com/paper/1908.07490/) (2019)
- [Generating Diverse High-Fidelity Images with VQ-VAE-2](https://mlsanity.com/paper/1906.00446/) (2019)
- [CogView: Mastering Text-to-Image Generation via Transformers](https://mlsanity.com/paper/2105.13290/) (2021)
- [Vector-quantized Image Modeling with Improved VQGAN](https://mlsanity.com/paper/2110.04627/) (2021)
- [ClipCap: CLIP Prefix for Image Captioning](https://mlsanity.com/paper/2111.09734/) (2021)
- [Hierarchical Question-Image Co-Attention for Visual Question Answering](https://mlsanity.com/paper/1606.00061/) (2016)
- [GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium](https://mlsanity.com/paper/1706.08500/) (2017)
- [SPAE: Semantic Pyramid AutoEncoder for Multimodal Generation with Frozen LLMs](https://mlsanity.com/paper/2306.17842/) (2023)
- [A Formal Evaluation of PSNR as Quality Measurement Parameter for Image Segmentation Algorithms](https://mlsanity.com/paper/1605.07116/) (2016)
- [Rethinking the Objectives of Vector-Quantized Tokenizers for Image Synthesis](https://mlsanity.com/paper/2212.03185/) (2022)
- [Language Quantized AutoEncoders: Towards Unsupervised Text-Image Alignment](https://mlsanity.com/paper/2302.00902/) (2023)
- [Beyond Text: Frozen Large Language Models in Visual Signal Comprehension](https://mlsanity.com/paper/2403.07874/) (2024)

## Built on by (most cited first, 10 of 10)

- [Language-Guided Image Tokenization for Generation](https://mlsanity.com/paper/2412.05796/) (2024)
- [Factorized Visual Tokenization and Generation](https://mlsanity.com/paper/2411.16681/) (2024)
- [Discrete Tokenization for Multimodal LLMs: A Comprehensive Survey](https://mlsanity.com/paper/2507.22920/) (2025)
- [Towards Improved Text-Aligned Codebook Learning: Multi-Hierarchical Codebook-Text Alignment with Long Text](https://mlsanity.com/paper/2503.01261/) (2025)
- [PyraTok: Language-Aligned Pyramidal Tokenizer for Video Understanding and Generation](https://mlsanity.com/paper/2601.16210/) (2026)
- [Improved Masked Image Generation with Knowledge-Augmented Token Representations](https://mlsanity.com/paper/2511.12032/) (2025)
- [Dual Codebook VQ: Enhanced Image Reconstruction with Reduced Codebook Size](https://mlsanity.com/paper/2503.10832/) (2025)
- [ArcVQ-VAE: A Spherical Vector Quantization Framework with ArcCosine Additive Margin](https://mlsanity.com/paper/2605.13517/) (2026)
- [Head-Aware Key-Value Compression for Efficient Autoregressive Image Generation](https://mlsanity.com/paper/2605.20600/) (2026)
- [Codebook-Guided Cross-Modal Knowledge Distillation for Structurally Heterogeneous Features](https://mlsanity.com/paper/2609.37243/) (2026)

## Abstract (arXiv)

Vector quantization (VQ) is a key technique in high-resolution and high-fidelity image synthesis, which aims to learn a codebook to encode an image with a sequence of discrete codes and then generate an image in an auto-regression manner. Although existing methods have shown superior performance, most methods prefer to learn a single-modal codebook (e.g., image), resulting in suboptimal performance when the codebook is applied to multi-modal downstream tasks (e.g., text-to-image, image captioning) due to the existence of modal gaps. In this paper, we propose a novel language-guided codebook learning framework, called LG-VQ, which aims to learn a codebook that can be aligned with the text to improve the performance of multi-modal downstream tasks. Specifically, we first introduce pre-trained text semantics as prior knowledge, then design two novel alignment modules (i.e., Semantic Alignment Module, and Relationship Alignment Module) to transfer such prior knowledge into codes for achieving codebook text alignment. In particular, our LG-VQ method is model-agnostic, which can be easily integrated into existing VQ models. Experimental results show that our method achieves superior performance on reconstruction and various multi-modal downstream tasks.