mlsanityData to 7 Oct 2026

Liquid: Language Models are Scalable and Unified Multi-modal Generators

arXiv 2412.04332 ↗First posted 5 Dec 2024cs.CVcited by 62

Not on the map. Why?

Abstract (arXiv)

We present Liquid, an auto-regressive generation paradigm that seamlessly integrates visual comprehension and generation by tokenizing images into discrete codes and learning these code embeddings alongside text tokens within a shared feature space for both vision and language. Unlike previous multimodal large language model (MLLM), Liquid achieves this integration using a single large language model (LLM), eliminating the need for external pretrained visual embeddings such as CLIP. For the first time, Liquid uncovers a scaling law that performance drop unavoidably brought by the unified training of visual and language tasks diminishes as the model size increases. Furthermore, the unified token space enables visual generation and comprehension tasks to mutually enhance each other, effectively removing the typical interference seen in earlier models. We show that existing LLMs can serve as strong foundations for Liquid, saving 100x in training costs while outperforming Chameleon in multimodal capabilities and maintaining language performance comparable to mainstream LLMs like LLAMA2. Liquid also outperforms models like SD v2.1 and SD-XL (FID of 5.47 on MJHQ-30K), excelling in both vision-language and text-only tasks. This work demonstrates that LLMs such as Qwen2.5 and GEMMA2 are powerful multimodal generators, offering a scalable solution for enhancing both vision-language understanding and generation. The code and models will be released at https://github.com/FoundationVision/Liquid.

Key facts. Cited by 62 papers in the map (data through 7 Oct 2026). Builds on 66 of its 88 references that are in the map. First posted on arXiv 5 Dec 2024.

Builds on

66 of 88 references.

References · 66

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2024The Llama 3 Herd of Modelscited by 14,192
  6. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  7. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  8. 2023Visual Instruction Tuningcited by 7,792
  9. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  10. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  11. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  12. 2022Classifier-Free Diffusion Guidancecited by 5,717
38 more
  1. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  2. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  3. 2017Neural Discrete Representation Learningcited by 4,650
  4. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  5. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  6. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  7. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,290
  8. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  9. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  10. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
  11. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  12. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  13. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  14. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  15. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  16. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
  17. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
  18. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,928
  19. 2019Towards VQA Models That Can Readcited by 1,696
  20. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
  21. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,624
  22. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,287
  23. 2024Qwen2.5-Coder Technical Reportcited by 1,151
  24. 2024Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Predictioncited by 1,005
  25. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captionscited by 1,000
  26. 2024Chameleon: Mixed-Modal Early-Fusion Foundation Modelscited by 904
  27. 2024Emu3: Next-Token Prediction is All You Needcited by 761
  28. 2024Autoregressive Model Beats Diffusion: Llama for Scalable Image Generationcited by 760
  29. 2024Show-o: One Single Transformer to Unify Multimodal Understanding and Generationcited by 734
  30. 2021CogView: Mastering Text-to-Image Generation via Transformerscited by 691
  31. 2023VILA: On Pre-training for Visual Language Modelscited by 686
  32. 2023Baichuan 2: Open Large-scale Language Modelscited by 664
  33. 2024Transfusion: Predict the Next Token and Diffuse Images with One Multi-Modal Modelcited by 468
  34. 2024Evaluating Text-to-Visual Generation with Image-to-Text Generationcited by 465
  35. 2023LMSYS-Chat-1M: A Large-Scale Real-World LLM Conversation Datasetcited by 449
  36. 2024Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and Generationcited by 447
  37. 2023Generative Multimodal Models are In-Context Learnerscited by 446
  38. 2024DataComp-LM: In search of the next generation of training sets for language modelscited by 381

50 most cited of 66 All 66 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 31; 2026: 31.

62 papers.

Sort:

By area:

  1. 2025UniTok: A Unified Tokenizer for Visual Generation and Understandingcited by 148
  2. 2025LongCat-Image Technical Reportcited by 92
  3. 2025Lumina-DiMOO: An Omni Diffusion Large Language Model for Multi-Modal Generation and Understandingcited by 83
  4. 2025VARGPT: Unified Understanding and Generation in a Visual Autoregressive Multimodal Large Language Modelcited by 44
  5. 2025InfinityStar: Unified Spacetime AutoRegressive Modeling for Visual Generationcited by 43
  6. 2025DualToken: Towards Unifying Visual Understanding and Generation with Dual Visual Vocabulariescited by 42
  7. 2025Omni-Video: Democratizing Unified Video Understanding and Generationcited by 39
  8. 2025RealUnify: Do Unified Models Truly Benefit from Unification? A Comprehensive Benchmarkcited by 33
  9. 2025Selftok: Discrete Visual Tokens of Autoregression, by Diffusion, and for Reasoningcited by 31
  10. 2025UniFlow: A Unified Pixel Flow Tokenizer for Visual Understanding and Generationcited by 23
  11. 2025Personalized Image Generation with Deep Generative Models: A Decade Surveycited by 20
  12. 2025Uni-X: Mitigating Modality Conflict with a Two-End-Separated Architecture for Unified Multimodal Modelscited by 18
38 more
  1. 2025AutoNeural: Co-Designing Vision-Language Models for NPU Inferencecited by 17
  2. 2025GIR-Bench: Versatile Benchmark for Generating Images with Reasoningcited by 15
  3. 2025Omni-View: Unlocking How Generation Facilitates Understanding in Unified 3D Model based on Multiview imagescited by 15
  4. 2025Cross-Task Generalization Between Understanding and Generation in Unified Vision-Language Models: A Controlled Studycited by 14
  5. 2026NextFlow: Unified Sequential Modeling Activates Multimodal Understanding and Generationcited by 13
  6. 2025UALM: Unified Audio Language Model for Understanding, Generation and Reasoningcited by 10
  7. 2025UniGame: Turning a Unified Multimodal Model Into Its Own Adversarycited by 9
  8. 2025Does Understanding Inform Generation in Unified Multimodal Models? From Analysis to Path Forwardcited by 9
  9. 2025COOPER: A Unified Model for Cooperative Perception and Reasoning in Spatial Intelligencecited by 9
  10. 2025Turning Internal Gap into Self-Improvement: Promoting the Generation-Understanding Unification in MLLMscited by 8
  11. 2026Unified Thinker: A General Reasoning Modular Core for Image Generationcited by 5
  12. 2026Self-Improving World Modelling with Latent Actionscited by 5
  13. 2025UniMoD: Efficient Unified Multimodal Transformers with Mixture-of-Depthscited by 4
  14. 2025Growing Visual Generative Capacity for Pre-Trained MLLMscited by 4
  15. 2026UniDDT: Unifying Multimodal Understanding and Generation with Decoupled Diffusion Transformercited by 4
  16. 2025ProtTeX: Structure-In-Context Reasoning and Editing of Proteins with Large Language Modelscited by 3
  17. 2025From Prediction to Perfection: Introducing Refinement to Autoregressive Image Generationcited by 3
  18. 2025A High-Quality Dataset and Reliable Evaluation for Interleaved Image-Text Generationcited by 3
  19. 2026UniX: Unifying Autoregression and Diffusion for Chest X-Ray Understanding and Generationcited by 3
  20. 2026Steering Visual Generation in Unified Multimodal Models with Understanding Supervisioncited by 3
  21. 2025Can VLMs Predict Future States? Bootstrapping World Models from Inverse Dynamicscited by 2
  22. 2026Intend, Reflect, Refine: An Adaptive Multimodal Reflection Framework for Autonomous Drivingcited by 2
  23. 2025Discrete Diffusion Models with MLLMs for Unified Medical Multimodal Generationcited by 1
  24. 2025Wave-Particle (Continuous-Discrete) Dualistic Visual Tokenization for Unified Understanding and Generationcited by 1
  25. 2026Pseudo-Unification: Entropy Probing Reveals Divergent Information Patterns in Unified Multimodal Modelscited by 1
  26. 2026WinTok: A Win-Win Hybrid Tokenizer via Decomposing Visual Understanding and Generation with Transferable Tokenscited by 1
  27. 2026On the Limits of Token Reduction for Efficient Unified Vision Language Trainingcited by 1
  28. 2026ARM: An AutoRegressive Large Multimodal Model with Unified Discrete Representationscited by 1
  29. 2026ThinkingVLA: Interleaved Vision and Language Reasoning for Robotic Manipulationcited by 1
  30. 2026Obliviate: Erasing Concepts from Autoregressive Image Generation Modelscited by 1
  31. 2026Transferability Between Understanding and Generation in Unified Multimodal Modelscited by 1
  32. 2026DeltaV: Thinking with Visual State Updates in Unified Large Multimodal Modelscited by 1
  33. 2026Towards Physics of Multimodal Pretraining: Knowledge Flow, Modality Synergy, Early Unification, and Recipescited by 1
  34. 2025EVLP:Learning Unified Embodied Vision-Language Planner with Reinforced Supervised Fine-Tuningcited by 0
  35. 2025Viper-F1: Fast and Fine-Grained Multimodal Understanding with Cross-Modal State-Space Modulationcited by 0
  36. 2026FedUMM: A General Framework for Federated Learning with Unified Multimodal Modelscited by 0
  37. 2026PnP-U3D: Plug-and-Play 3D Framework Bridging Autoregression and Diffusion for Unified Understanding and Generationcited by 0
  38. 2026Synergizing Understanding and Generation with Interleaved Analyzing-Drafting Thinkingcited by 0

50 most cited of 62 All 62 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024LMFusion: Adapting Pretrained Language Models for Multimodal Generation
  2. 2025Vision as a Dialect: Unifying Visual Understanding and Generation via Text-Aligned Representations
  3. 2025MANZANO: A Simple and Scalable Unified Multimodal Model with a Hybrid Vision Tokenizer
  4. 2025Generative Multimodal Pretraining with Discrete Diffusion Timestep Tokens
  5. 2026Generation as Auxiliary Supervision: Enhancing Visual Understanding at Zero Inference Overhead via Decoupled Embedding Prediction
  6. 2026Multimodal Flow: Unified Flow Modeling of Language and Vision in Embedding Spaces
  7. 2025Perceiving Beyond Language Priors: Enhancing Visual Comprehension and Attention in Multimodal Models
  8. 2023Generating Images with Multimodal Language Models