mlsanityData to 7 Oct 2026

VILA-U: a Unified Foundation Model Integrating Visual Understanding and Generation

arXiv 2409.04429 ↗First posted 6 Sep 2024cs.CV cs.LGcited by 290

On the map: Diffusion models → Image generation, text-to-image and editing

Abstract (arXiv)

VILA-U is a Unified foundation model that integrates Video, Image, Language understanding and generation. Traditional visual language models (VLMs) use separate modules for understanding and generating visual content, which can lead to misalignment and increased complexity. In contrast, VILA-U employs a single autoregressive next-token prediction framework for both tasks, eliminating the need for additional components like diffusion models. This approach not only simplifies the model but also achieves near state-of-the-art performance in visual language understanding and generation. The success of VILA-U is attributed to two main factors: the unified vision tower that aligns discrete visual tokens with textual inputs during pretraining, which enhances visual perception, and autoregressive image generation can achieve similar quality as diffusion models with high-quality dataset. This allows VILA-U to perform comparably to more complex models using a fully token-based autoregressive framework.

Key facts. Cited by 290 papers in the map (data through 7 Oct 2026). Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models. Builds on 63 of its 81 references that are in the map. First posted on arXiv 6 Sep 2024.

Builds on

63 of 81 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024DEEM: Diffusion Models Serve as the Eyes of Large Language Models for Image Perception24 May
  2. 2024Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Prediction3 Apr
  3. 2024Playground v2.5: Three Insights towards Enhancing Aesthetic Quality in Text-to-Image Generation27 Feb

Other references · 60

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  4. 2022Training language models to follow instructions with human feedbackcited by 15,712
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  6. 2023Visual Instruction Tuningcited by 7,792
  7. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  8. 2022Classifier-Free Diffusion Guidancecited by 5,717
  9. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  10. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  11. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  12. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,290
38 more
  1. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
  2. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  3. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  4. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  5. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  6. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
  7. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  8. 2024CogVideoX: Text-to-Video Diffusion Models with An Expert Transformercited by 2,198
  9. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  10. 2023VBench: Comprehensive Benchmark Suite for Video Generative Modelscited by 1,755
  11. 2019Towards VQA Models That Can Readcited by 1,696
  12. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
  13. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,461
  14. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,385
  15. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,221
  16. 2021Perceiver: General Perception with Iterative Attentioncited by 1,089
  17. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,047
  18. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
  19. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captionscited by 1,000
  20. 2022CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformerscited by 973
  21. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehensioncited by 929
  22. 2023VideoChat: Chat-Centric Video Understandingcited by 851
  23. 2024Autoregressive Model Beats Diffusion: Llama for Scalable Image Generationcited by 760
  24. 2024Show-o: One Single Transformer to Unify Multimodal Understanding and Generationcited by 734
  25. 2022Autoregressive Image Generation using Residual Quantizationcited by 706
  26. 2023CogAgent: A Visual Language Model for GUI Agentscited by 692
  27. 2021Vector-quantized Image Modeling with Improved VQGANcited by 656
  28. 2024Evaluating Text-to-Visual Generation with Image-to-Text Generationcited by 465
  29. 2023Generative Multimodal Models are In-Context Learnerscited by 446
  30. 2023Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruningcited by 408
  31. 2024Mini-Gemini: Mining the Potential of Multi-modality Vision Language Modelscited by 361
  32. 2024OpenVid-1M: A Large-Scale High-Quality Dataset for Text-to-video Generationcited by 332
  33. 2024SEED-X: Multimodal Models with Unified Multi-granularity Comprehension and Generationcited by 328
  34. 2023Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Actioncited by 301
  35. 2023InternLM-XComposer: A Vision-Language Large Model for Advanced Text-image Comprehension and Compositioncited by 297
  36. 2023DreamLLM: Synergistic Multimodal Comprehension and Creationcited by 296
  37. 2023PaLI-X: On Scaling up a Multilingual Vision and Language Modelcited by 254
  38. 2024AnyGPT: Unified Multimodal LLM with Discrete Sequence Modelingcited by 213

50 most cited of 60 All 60 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 31; 2025: 158; 2026: 101.

290 papers.

Sort:

By area:

  1. 2025Emerging Properties in Unified Multimodal Pretrainingcited by 855
  2. 2025Cosmos World Foundation Model Platform for Physical AIcited by 833
  3. 2024Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and Generationcited by 447
  4. 2025CoT-VLA: Visual Chain-of-Thought Reasoning for Vision-Language-Action Modelscited by 440
  5. 2024From Slow Bidirectional to Fast Autoregressive Video Diffusion Modelscited by 401
  6. 2025Show-o2: Improved Native Unified Multimodal Modelscited by 242
  7. 2025MMaDA: Multimodal Large Diffusion Language Modelscited by 241
  8. 2025Transfer between Modalities with MetaQueriescited by 231
  9. 2024NaVILA: Legged Robot Vision-Language-Action Model for Navigationcited by 201
  10. 2024TokenFlow: Unified Image Tokenizer for Multimodal Understanding and Generationcited by 200
  11. 2025WISE: A World Knowledge-Informed Semantic Evaluation for Text-to-Image Generationcited by 196
  12. 2025Thinking with Images for Multimodal Reasoning: Foundations, Methods, and Future Frontierscited by 193
38 more
  1. 2025HybridVLA: Collaborative Diffusion and Autoregression in a Unified Vision-Language-Action Modelcited by 176
  2. 2025T2I-R1: Reinforcing Image Generation with Collaborative Semantic-level and Token-level CoTcited by 162
  3. 2025UniTok: A Unified Tokenizer for Visual Generation and Understandingcited by 148
  4. 2024HART: Efficient Visual Generation with Hybrid Autoregressive Transformercited by 128
  5. 2025Spatial Forcing: Implicit Spatial Representation Alignment for Vision-language-action Modelcited by 125
  6. 2024LMFusion: Adapting Pretrained Language Models for Multimodal Generationcited by 120
  7. 2025Mogao: An Omni Foundation Model for Interleaved Multi-Modal Generationcited by 108
  8. 2025X-Omni: Reinforcement Learning Makes Discrete Autoregressive Image Generative Models Great Againcited by 105
  9. 2024UniReal: Universal Image Generation and Editing via Learning Real-world Dynamicscited by 100
  10. 2025Envisioning Beyond the Pixels: Benchmarking Reasoning-Informed Visual Editingcited by 97
  11. 2024ImageFolder: Autoregressive Image Generation with Folded Tokenscited by 93
  12. 2025Masked Autoencoders Are Effective Tokenizers for Diffusion Modelscited by 93
  13. 2024LLaMA-Mesh: Unifying 3D Mesh Generation with Language Modelscited by 87
  14. 2025SimpleAR: Pushing the Frontier of Autoregressive Visual Generation through Pretraining, SFT, and RLcited by 85
  15. 2025Lumina-DiMOO: An Omni Diffusion Large Language Model for Multi-Modal Generation and Understandingcited by 83
  16. 2025Unified Multimodal Understanding and Generation Models: Advances, Challenges, and Opportunitiescited by 81
  17. 2024Dual Diffusion for Unified Image Generation and Understandingcited by 76
  18. 2025Harmonizing Visual Representations for Unified Multimodal Understanding and Generationcited by 74
  19. 2025Baichuan-Audio: A Unified Framework for End-to-End Speech Interactioncited by 68
  20. 2024ILLUME: Illuminating Your LLMs to See, Draw, and Self-Enhancecited by 64
  21. 2025RIFLEx: A Free Lunch for Length Extrapolation in Video Diffusion Transformerscited by 63
  22. 2024Diffusion-VLA: Generalizable and Interpretable Robot Foundation Model via Self-Generated Reasoningcited by 57
  23. 2025GPT-ImgEval: A Comprehensive Benchmark for Diagnosing GPT4o in Image Generationcited by 57
  24. 2024SoftVQ-VAE: Efficient 1-Dimensional Continuous Tokenizercited by 55
  25. 2025GigaTok: Scaling Visual Tokenizers to 3 Billion Parameters for Autoregressive Image Generationcited by 55
  26. 2025ILLUME+: Illuminating Unified MLLM with Dual Visual Tokenization and Diffusion Refinementcited by 53
  27. 2025Muddit: Liberating Generation Beyond Text-to-Image with a Unified Discrete Diffusion Modelcited by 52
  28. 2025TokLIP: Marry Visual Tokens to CLIP for Multimodal Comprehension and Generationcited by 50
  29. 2025OneCAT: Decoder-Only Auto-Regressive Model for Unified Understanding and Generationcited by 50
  30. 2025UniToken: Harmonizing Multimodal Understanding and Generation through Unified Visual Encodingcited by 49
  31. 2026Scaling Text-to-Image Diffusion Transformers with Representation Autoencoderscited by 49
  32. 2025Reconstruction Alignment Improves Unified Multimodal Modelscited by 48
  33. 2025T2ISafety: Benchmark for Assessing Fairness, Toxicity, and Privacy in Image Generationcited by 47
  34. 2024The Language of Motion: Unifying Verbal and Non-verbal Language of 3D Human Motioncited by 46
  35. 2025FUDOKI: Discrete Flow-based Unified Understanding and Generation via Kinetic-Optimal Velocitiescited by 46
  36. 2024ACDiT: Interpolating Autoregressive Conditional Modeling and Diffusion Transformercited by 45
  37. 2025OpenUni: A Simple Baseline for Unified Multimodal Understanding and Generationcited by 45
  38. 2024MUSE-VL: Modeling Unified VLM through Semantic Discrete Encodingcited by 44

50 most cited of 290 All 290 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024VILA2: VILA Augmented VILA
  2. 2023VILA: On Pre-training for Visual Language Models
  3. 2025VARGPT: Unified Understanding and Generation in a Visual Autoregressive Multimodal Large Language Model
  4. 2023u-LLaVA: Unifying Multi-Modal Tasks via Large Language Model
  5. 2025UniHetero: Could Generation Enhance Understanding for Vision-Language-Model at Large Data Scale?
  6. 2023Measuring Progress in Fine-grained Vision-and-Language Understanding
  7. 2024An Introduction to Vision-Language Modeling
  8. 2023VLIS: Unimodal Language Models Guide Multimodal Language Generation