mlsanityData to 7 Oct 2026

QLIP: Text-Aligned Visual Tokenization Unifies Auto-Regressive Multimodal Understanding and Generation

arXiv 2502.05178 ↗First posted 7 Feb 2025cs.CVcited by 44

Not on the map. Why?

Abstract (arXiv)

We introduce Quantized Language-Image Pretraining (QLIP), a visual tokenization method that combines state-of-the-art reconstruction quality with state-of-the-art zero-shot image understanding. QLIP trains a binary-spherical-quantization-based autoencoder with reconstruction and language-image alignment objectives. We are the first to show that the two objectives do not need to be at odds. We balance the two loss terms dynamically during training and show that a two-stage training pipeline effectively mixes the large-batch requirements of image-language pre-training with the memory bottleneck imposed by the reconstruction objective. We validate the effectiveness of QLIP for multimodal understanding and text-conditioned image generation with a single model. Specifically, QLIP serves as a drop-in replacement for the visual encoder for LLaVA and the image tokenizer for LlamaGen with comparable or even better performance. Finally, we demonstrate that QLIP enables a unified mixed-modality auto-regressive model for understanding and generation.

Key facts. Cited by 44 papers in the map (data through 7 Oct 2026). First posted on arXiv 7 Feb 2025.

Builds on

0 of 0 references.

No references here yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 20; 2026: 24.

44 papers.

Sort:

By area:

  1. 2025Unified Multimodal Understanding and Generation Models: Advances, Challenges, and Opportunitiescited by 81
  2. 2025TokLIP: Marry Visual Tokens to CLIP for Multimodal Comprehension and Generationcited by 50
  3. 2025Vision as a Dialect: Unifying Visual Understanding and Generation via Text-Aligned Representationscited by 44
  4. 2025DualToken: Towards Unifying Visual Understanding and Generation with Dual Visual Vocabulariescited by 42
  5. 2025JarvisArt: Liberating Human Artistic Creativity via an Intelligent Photo Retouching Agentcited by 39
  6. 2025AlignTok: Aligning Visual Foundation Encoders to Tokenizers for Diffusion Modelscited by 35
  7. 2026LongCat-Next: Lexicalizing Modalities as Discrete Tokenscited by 30
  8. 2026SenseNova-U1: Unifying Multimodal Understanding and Generation with NEO-unify Architecturecited by 27
  9. 2025UniFlow: A Unified Pixel Flow Tokenizer for Visual Understanding and Generationcited by 23
  10. 2025SemHiTok: A Unified Image Tokenizer via Semantic-Guided Hierarchical Codebook for Multimodal Understanding and Generationcited by 21
  11. 2025VARGPT-v1.1: Improve Visual Autoregressive Large Unified Model via Iterative Instruction Tuning and Reinforcement Learningcited by 21
  12. 2025MoCa: Modality-aware Continual Pre-training Makes Better Bidirectional Multimodal Embeddingscited by 20
32 more
  1. 2025GeoUni: A Unified Model for Generating Geometry Diagrams, Problems and Problem Solutionscited by 11
  2. 2025WeTok: Powerful Discrete Tokenization for High-Fidelity Visual Reconstructioncited by 9
  3. 2025VQRAE: Representation Quantization Autoencoders for Multimodal Understanding, Generation and Reconstructioncited by 9
  4. 2025Locality-aware Parallel Decoding for Efficient Autoregressive Image Generationcited by 8
  5. 2025Discrete Tokenization for Multimodal LLMs: A Comprehensive Surveycited by 7
  6. 2025Spherical Leech Quantization for Visual Tokenization and Generationcited by 6
  7. 2025End-to-End Vision Tokenizer Tuningcited by 5
  8. 2025UniModel: A Visual-Only Framework for Unified Multimodal Understanding and Generationcited by 3
  9. 2026Unified Multimodal Autoregressive Modeling with Shared Context-Visual Tokenizer is Key to Unificationcited by 3
  10. 2026EvoTok: A Unified Image Tokenizer via Residual Latent Evolution for Visual Understanding and Generationcited by 2
  11. 2026HYDRA: Unifying Multi-modal Generation and Understanding via Representation-Harmonized Tokenizationcited by 2
  12. 2025Wave-Particle (Continuous-Discrete) Dualistic Visual Tokenization for Unified Understanding and Generationcited by 1
  13. 2026Semantic One-Dimensional Tokenizer for Image Reconstruction and Generationcited by 1
  14. 2026WinTok: A Win-Win Hybrid Tokenizer via Decomposing Visual Understanding and Generation with Transferable Tokenscited by 1
  15. 2026ViQ: Text-Aligned Visual Quantized Representations at Any Resolutioncited by 1
  16. 2026Towards Physics of Multimodal Pretraining: Knowledge Flow, Modality Synergy, Early Unification, and Recipescited by 1
  17. 2025UniCode2: Cascaded Large-scale Codebooks for Unified Multimodal Understanding and Generationcited by 0
  18. 2026A Vision for Multisensory Intelligence: Sensing, Science, and Synergycited by 0
  19. 2026PnP-U3D: Plug-and-Play 3D Framework Bridging Autoregression and Diffusion for Unified Understanding and Generationcited by 0
  20. 2026UniWeTok: An Unified Binary Tokenizer with Codebook Size mathit2128 for Unified Multimodal Large Language Modelcited by 0
  21. 2026Decoupling Vision and Language: Codebook Anchored Visual Adaptationcited by 0
  22. 2026Unified Pix Token And Word Token Generative Language Modelcited by 0
  23. 2026NSVQ: Mitigating Codebook Collapse by Stabilizing Encoder Drift in Vector Quantizationcited by 0
  24. 2026dRAE: Representation Autoencoder with Hyper-Spherical Codescited by 0
  25. 2026Twins: Learn to Predict Unified Representations with Focal Losscited by 0
  26. 2026Multimodal Alignment Through Joint Kernel Entropic Gromov--Wasserstein Optimal Transportcited by 0
  27. 2026UniSpace: Unified Visual Representation and Scalable Multimodal Modelingcited by 0
  28. 2026Expressivity In Multimodal Contrastive Learningcited by 0
  29. 2026Studying Image Tokenizers as Visual Languages in Unified Multimodal Modelscited by 0
  30. 2026FLAT: Resampling Image and Text into 1D Flexible-Length Aligned Transmodal Tokens for Retrieval and Generationcited by 0
  31. 2026PixelUMM: Encoder-Free Unified Image and Video Understanding and Generationcited by 0
  32. 2026QuadTok: Quadtree Visual Tokenizer for Autoregressive Image Generationcited by 0