# QLIP: Text-Aligned Visual Tokenization Unifies Auto-Regressive Multimodal Understanding and Generation

arXiv: https://arxiv.org/abs/2502.05178 · HTML: https://mlsanity.com/paper/2502.05178/

## Key facts

- Cited by 45 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Feb 2025.

## Built on by (most cited first, 30 of 45)

- [Unified Multimodal Understanding and Generation Models: Advances, Challenges, and Opportunities](https://mlsanity.com/paper/2505.02567/) (2025)
- [TokLIP: Marry Visual Tokens to CLIP for Multimodal Comprehension and Generation](https://mlsanity.com/paper/2505.05422/) (2025)
- [Vision as a Dialect: Unifying Visual Understanding and Generation via Text-Aligned Representations](https://mlsanity.com/paper/2506.18898/) (2025)
- [DualToken: Towards Unifying Visual Understanding and Generation with Dual Visual Vocabularies](https://mlsanity.com/paper/2503.14324/) (2025)
- [JarvisArt: Liberating Human Artistic Creativity via an Intelligent Photo Retouching Agent](https://mlsanity.com/paper/2506.17612/) (2025)
- [AlignTok: Aligning Visual Foundation Encoders to Tokenizers for Diffusion Models](https://mlsanity.com/paper/2509.25162/) (2025)
- [LongCat-Next: Lexicalizing Modalities as Discrete Tokens](https://mlsanity.com/paper/2603.27538/) (2026)
- [SenseNova-U1: Unifying Multimodal Understanding and Generation with NEO-unify Architecture](https://mlsanity.com/paper/2605.12500/) (2026)
- [UniFlow: A Unified Pixel Flow Tokenizer for Visual Understanding and Generation](https://mlsanity.com/paper/2510.10575/) (2025)
- [SemHiTok: A Unified Image Tokenizer via Semantic-Guided Hierarchical Codebook for Multimodal Understanding and Generation](https://mlsanity.com/paper/2503.06764/) (2025)
- [VARGPT-v1.1: Improve Visual Autoregressive Large Unified Model via Iterative Instruction Tuning and Reinforcement Learning](https://mlsanity.com/paper/2504.02949/) (2025)
- [MoCa: Modality-aware Continual Pre-training Makes Better Bidirectional Multimodal Embeddings](https://mlsanity.com/paper/2506.23115/) (2025)
- [GeoUni: A Unified Model for Generating Geometry Diagrams, Problems and Problem Solutions](https://mlsanity.com/paper/2504.10146/) (2025)
- [Locality-aware Parallel Decoding for Efficient Autoregressive Image Generation](https://mlsanity.com/paper/2507.01957/) (2025)
- [VQRAE: Representation Quantization Autoencoders for Multimodal Understanding, Generation and Reconstruction](https://mlsanity.com/paper/2511.23386/) (2025)
- [WeTok: Powerful Discrete Tokenization for High-Fidelity Visual Reconstruction](https://mlsanity.com/paper/2508.05599/) (2025)
- [Discrete Tokenization for Multimodal LLMs: A Comprehensive Survey](https://mlsanity.com/paper/2507.22920/) (2025)
- [Spherical Leech Quantization for Visual Tokenization and Generation](https://mlsanity.com/paper/2512.14697/) (2025)
- [End-to-End Vision Tokenizer Tuning](https://mlsanity.com/paper/2505.10562/) (2025)
- [Unified Multimodal Autoregressive Modeling with Shared Context-Visual Tokenizer is Key to Unification](https://mlsanity.com/paper/2606.18249/) (2026)
- [UniModel: A Visual-Only Framework for Unified Multimodal Understanding and Generation](https://mlsanity.com/paper/2511.16917/) (2025)
- [Towards Physics of Multimodal Pretraining: Knowledge Flow, Modality Synergy, Early Unification, and Recipes](https://mlsanity.com/paper/2608.05000/) (2026)
- [EvoTok: A Unified Image Tokenizer via Residual Latent Evolution for Visual Understanding and Generation](https://mlsanity.com/paper/2603.12108/) (2026)
- [HYDRA: Representation Harmonized Tokenization for Multimodal Generation and Understanding](https://mlsanity.com/paper/2603.15228/) (2026)
- [Semantic One-Dimensional Tokenizer for Image Reconstruction and Generation](https://mlsanity.com/paper/2603.16373/) (2026)
- [Wave-Particle (Continuous-Discrete) Dualistic Visual Tokenization for Unified Understanding and Generation](https://mlsanity.com/paper/2511.01593/) (2025)
- [WinTok: A Win-Win Hybrid Tokenizer via Decomposing Visual Understanding and Generation with Transferable Tokens](https://mlsanity.com/paper/2605.18115/) (2026)
- [ViQ: Text-Aligned Visual Quantized Representations at Any Resolution](https://mlsanity.com/paper/2606.27313/) (2026)
- [Twins: Learn to Predict Unified Representations with Focal Loss](https://mlsanity.com/paper/2607.22531/) (2026)
- [UniSpace: Unified Visual Representation and Scalable Multimodal Modeling](https://mlsanity.com/paper/2608.08676/) (2026)

## Abstract (arXiv)

We introduce Quantized Language-Image Pretraining (QLIP), a visual tokenization method that combines state-of-the-art reconstruction quality with state-of-the-art zero-shot image understanding. QLIP trains a binary-spherical-quantization-based autoencoder with reconstruction and language-image alignment objectives. We are the first to show that the two objectives do not need to be at odds. We balance the two loss terms dynamically during training and show that a two-stage training pipeline effectively mixes the large-batch requirements of image-language pre-training with the memory bottleneck imposed by the reconstruction objective. We validate the effectiveness of QLIP for multimodal understanding and text-conditioned image generation with a single model. Specifically, QLIP serves as a drop-in replacement for the visual encoder for LLaVA and the image tokenizer for LlamaGen with comparable or even better performance. Finally, we demonstrate that QLIP enables a unified mixed-modality auto-regressive model for understanding and generation.