mlsanityData to 14 Oct 2026

TokLIP: Marry Visual Tokens to CLIP for Multimodal Comprehension and Generation

arXiv 2505.05422 ↗First posted 8 May 2025cs.CV cs.AI cs.CLcited by 51

Not on the map. Why?

Abstract (arXiv)

Pioneering token-based works such as Chameleon and Emu3 have established a foundation for multimodal unification but face challenges of high training computational overhead and limited comprehension performance due to a lack of high-level semantics. In this paper, we introduce TokLIP, a visual tokenizer that enhances comprehension by semanticizing vector-quantized (VQ) tokens and incorporating CLIP-level semantics while enabling end-to-end multimodal autoregressive training with standard VQ tokens. TokLIP integrates a low-level discrete VQ tokenizer with a ViT-based token encoder to capture high-level continuous semantics. Unlike previous approaches (e.g., VILA-U) that discretize high-level features, TokLIP disentangles training objectives for comprehension and generation, allowing the direct application of advanced VQ tokenizers without the need for tailored quantization operations. Our empirical results demonstrate that TokLIP achieves exceptional data efficiency, empowering visual tokens with high-level semantic understanding while enhancing low-level generative capacity, making it well-suited for autoregressive Transformers in both comprehension and generation tasks. The code and models are available at https://github.com/TencentARC/TokLIP.

Key facts. Cited by 51 papers in the map (data through 14 Oct 2026). Builds on 76 of its 100 references that are in the map. First posted on arXiv 8 May 2025.

Builds on

76 of 100 references.

References · 76

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2023Visual Instruction Tuningcited by 7,869
  3. 2024Qwen2.5 Technical Reportcited by 4,942
  4. 2017Neural Discrete Representation Learningcited by 4,712
  5. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  6. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,356
  7. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
  8. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,253
  9. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
  10. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,719
  11. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  12. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,141
38 more
  1. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,080
  2. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,729
  3. 2013Fine-Grained Visual Classification of Aircraftcited by 1,726
  4. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,652
  5. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,555
  6. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,462
  7. 2019Generating Diverse High-Fidelity Images with VQ-VAE-2cited by 1,387
  8. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,239
  9. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,216
  10. 2025SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Featurescited by 1,188
  11. 2022Reproducible scaling laws for contrastive language-image learningcited by 1,173
  12. 2022MaskGIT: Masked Generative Image Transformercited by 1,099
  13. 2021CLIP-Adapter: Better Vision-Language Models with Feature Adapterscited by 1,026
  14. 2024Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Predictioncited by 1,023
  15. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehensioncited by 940
  16. 2024Chameleon: Mixed-Modal Early-Fusion Foundation Modelscited by 927
  17. 2025Cosmos World Foundation Model Platform for Physical AIcited by 864
  18. 2025Janus-Pro: Unified Multimodal Understanding and Generation with Data and Model Scalingcited by 812
  19. 2024Emu3: Next-Token Prediction is All You Needcited by 781
  20. 2024Autoregressive Model Beats Diffusion: Llama for Scalable Image Generationcited by 777
  21. 2024Show-o: One Single Transformer to Unify Multimodal Understanding and Generationcited by 752
  22. 2024DeepSeek-VL: Towards Real-World Vision-Language Understandingcited by 733
  23. 2022Autoregressive Image Generation using Residual Quantizationcited by 722
  24. 2021Vector-quantized Image Modeling with Improved VQGANcited by 658
  25. 2023DataComp: In search of the next generation of multimodal datasetscited by 644
  26. 2023Language Model Beats Diffusion -- Tokenizer is Key to Visual Generationcited by 640
  27. 2023NExT-GPT: Any-to-Any Multimodal LLMcited by 615
  28. 2024Transfusion: Predict the Next Token and Diffuse Images with One Multi-Modal Modelcited by 483
  29. 2024Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and Generationcited by 457
  30. 2023Generative Multimodal Models are In-Context Learnerscited by 446
  31. 2023EVA-02: A Visual Representation for Neon Genesiscited by 381
  32. 2024SEED-X: Multimodal Models with Unified Multi-granularity Comprehension and Generationcited by 331
  33. 2023DreamLLM: Synergistic Multimodal Comprehension and Creationcited by 297
  34. 2024VILA-U: a Unified Foundation Model Integrating Visual Understanding and Generationcited by 295
  35. 2023Demystifying CLIP Datacited by 270
  36. 2023Data Filtering Networkscited by 250
  37. 2025Transfer between Modalities with MetaQueriescited by 236
  38. 2024MetaMorph: Multimodal Understanding and Generation via Instruction Tuningcited by 206

50 most cited of 76 All 76 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 25; 2026: 26.

51 papers.

Sort:

By area:

  1. 2025Show-o2: Improved Native Unified Multimodal Modelscited by 250
  2. 2025UniLiP: Adapting CLIP for Unified Multimodal Understanding, Generation and Editingcited by 44
  3. 2025AlignTok: Aligning Visual Foundation Encoders to Tokenizers for Diffusion Modelscited by 36
  4. 2025TUNA: Taming Unified Visual Representations for Native Unified Multimodal Modelscited by 34
  5. 2026SenseNova-U1: Unifying Multimodal Understanding and Generation with NEO-unify Architecturecited by 32
  6. 2025MindOmni: Unleashing Reasoning Generation in Vision Language Models with RGPOcited by 30
  7. 2025Ming-UniVision: Joint Image Understanding and Generation with a Unified Continuous Tokenizercited by 28
  8. 2025From Perception to Cognition: A Survey of Vision-Language Interactive Reasoning in Multimodal Large Language Modelscited by 26
  9. 2025Quantization Meets dLLMs: A Systematic Study of Post-training Quantization for Diffusion LLMscited by 23
  10. 2025UniFlow: A Unified Pixel Flow Tokenizer for Visual Understanding and Generationcited by 23
  11. 2025Both Semantics and Reconstruction Matter: Making Representation Encoders Ready for Text-to-Image Generation and Editingcited by 22
  12. 2025Thinking with Camera: A Unified Multimodal Model for Camera-Centric Understanding and Generationcited by 19
38 more
  1. 2025Uni-X: Mitigating Modality Conflict with a Two-End-Separated Architecture for Unified Multimodal Modelscited by 18
  2. 2026Tuna-2: Pixel Embeddings Beat Vision Encoders for Multimodal Understanding and Generationcited by 18
  3. 2025PAROAttention: Pattern-Aware ReOrdering for Efficient Sparse and Quantized Attention in Visual Generation Modelscited by 17
  4. 2025LRQ-DiT: Log-Rotation Post-Training Quantization of Diffusion Transformers for Image and Video Generationcited by 12
  5. 2025Locality-aware Parallel Decoding for Efficient Autoregressive Image Generationcited by 10
  6. 2025Hierarchical LoRA MoE for Efficient CTR Model Scalingcited by 10
  7. 2025Harnessing Consistency for Robust Test-Time LLM Ensemblecited by 10
  8. 2025VQRAE: Representation Quantization Autoencoders for Multimodal Understanding, Generation and Reconstructioncited by 10
  9. 2025Discrete Tokenization for Multimodal LLMs: A Comprehensive Surveycited by 8
  10. 2026LatentUM: Unleashing the Potential of Interleaved Cross-Modal Reasoning via a Latent-Space Unified Modelcited by 8
  11. 2025MedREK: Retrieval-Based Editing for Medical LLMs with Key-Aware Promptscited by 5
  12. 2025CTR-Sink: Attention Sink for Language Models in Click-Through Rate Predictioncited by 4
  13. 2026PyraTok: Language-Aligned Pyramidal Tokenizer for Video Understanding and Generationcited by 4
  14. 2025AudioStory: Generating Long-Form Narrative Audio with Large Language Modelscited by 3
  15. 2025UniModel: A Visual-Only Framework for Unified Multimodal Understanding and Generationcited by 3
  16. 2026Predict, Don't Iterate: Efficient Adaptive-Length Infilling for Diffusion Language Modelscited by 3
  17. 2025Resolving Task Objective Conflicts in Unified Model via Task-Aware Mixture-of-Expertscited by 2
  18. 2026IMTBench: A Multi-Scenario Cross-Modal Collaborative Evaluation Benchmark for In-Image Machine Translationcited by 2
  19. 2026EvoTok: A Unified Image Tokenizer via Residual Latent Evolution for Visual Understanding and Generationcited by 2
  20. 2026HYDRA-X: Native Unified Multimodal Models with Holistic Visual Tokenizerscited by 2
  21. 2026IV-CoT: Implicit Visual Chain-of-Thought for Structure-Aware Text-to-Image Generationcited by 2
  22. 2026WinTok: A Win-Win Hybrid Tokenizer via Decomposing Visual Understanding and Generation with Transferable Tokenscited by 1
  23. 2026SPAR: Semantic-Pixel Self-Alignment and Adaptive Routing for Unified Multimodal Modelscited by 1
  24. 2026UniTranslator: A Unified Multi-modal Framework for End-to-end In-Image Machine Translationcited by 1
  25. 2026Twins: Learn to Predict Unified Representations with Focal Losscited by 1
  26. 2026UniSpace: Unified Visual Representation and Scalable Multimodal Modelingcited by 1
  27. 2026PixelUMM: Encoder-Free Unified Image and Video Understanding and Generationcited by 1
  28. 2025JEPA-T: Joint-Embedding Predictive Architecture with Text Fusion for Image Generationcited by 0
  29. 2026InfoTok: Information-Theoretic Regularization for Capacity-Constrained Shared Visual Tokenization in Unified MLLMscited by 0
  30. 2026Decoupling Vision and Language: Codebook Anchored Visual Adaptationcited by 0
  31. 2026InsightTok: Improving Text and Face Fidelity in Discrete Tokenization for Autoregressive Image Generationcited by 0
  32. 2026ProductWebGen: Benchmarking Multimodal Product Webpage Generationcited by 0
  33. 2026GroupVideo: Multi-Identity Customized Text-to-Video Generationcited by 0
  34. 2026dRAE: Representation Autoencoder with Hyper-Spherical Codescited by 0
  35. 2026Argus-Unified: Towards A Compact and Economical Unified Model for Image Understanding and Generationcited by 0
  36. 2026Puffin-World: Scaling a Unified Multimodal Model with Native 3D World Statescited by 0
  37. 2026Studying Image Tokenizers as Visual Languages in Unified Multimodal Modelscited by 0
  38. 2026FLAT: Resampling Image and Text into 1D Flexible-Length Aligned Transmodal Tokens for Retrieval and Generationcited by 0

50 most cited of 51 All 51 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024ViCToR: Improving Visual Comprehension via Token Reconstruction for Pretraining LMMs
  2. 2023Planting a SEED of Vision in Large Language Model
  3. 2024Liquid: Language Models are Scalable and Unified Multi-modal Generators
  4. 2025VITRIX-CLIPIN: Enhancing Fine-Grained Visual Understanding in CLIP via Instruction Editing Data and Long Captions
  5. 2026HYDRA: Representation Harmonized Tokenization for Multimodal Generation and Understanding
  6. 2024Auto-Encoding Morph-Tokens for Multimodal LLM
  7. 2024From Pixels to Tokens: Byte-Pair Encoding on Quantized Visual Modalities
  8. 2025VARGPT: Unified Understanding and Generation in a Visual Autoregressive Multimodal Large Language Model