mlsanityData to 14 Oct 2026

VARGPT: Unified Understanding and Generation in a Visual Autoregressive Multimodal Large Language Model

arXiv 2501.12327 ↗First posted 21 Jan 2025cs.CVcited by 44

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

We present VARGPT, a novel multimodal large language model (MLLM) that unifies visual understanding and generation within a single autoregressive framework. VARGPT employs a next-token prediction paradigm for visual understanding and a next-scale prediction paradigm for visual autoregressive generation. VARGPT innovatively extends the LLaVA architecture, achieving efficient scale-wise autoregressive visual generation within MLLMs while seamlessly accommodating mixed-modal input and output within a single model framework. Our VARGPT undergoes a three-stage unified training process on specially curated datasets, comprising a pre-training phase and two mixed visual instruction-tuning phases. The unified training strategy are designed to achieve alignment between visual and textual features, enhance instruction following for both understanding and generation, and improve visual generation quality, respectively. Despite its LLAVA-based architecture for multimodel understanding, VARGPT significantly outperforms LLaVA-1.5 across various vision-centric benchmarks, such as visual question-answering and reasoning tasks. Notably, VARGPT naturally supports capabilities in autoregressive visual generation and instruction-to-image synthesis, showcasing its versatility in both visual understanding and generation tasks. Project page is at: urlhttps://vargpt-1.github.io/

Key facts. Cited by 44 papers in the map (data through 14 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 58 of its 81 references that are in the map. First posted on arXiv 21 Jan 2025.

Builds on

58 of 81 references.

Closest ancestors · 9

Same direction, newest first

  1. 2024LMMs-Eval: Reality Check on the Evaluation of Large Multimodal Models17 Jul
  2. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks21 Dec
  3. 2023mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaboration7 Nov
  4. 2023Improved Baselines with Visual Instruction Tuning5 Oct
  5. 2023NExT-GPT: Any-to-Any Multimodal LLM11 Sep
  6. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehension30 Jul
  7. 2023MMBench: Is Your Multi-modal Model an All-around Player?12 Jul
  8. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models23 Jun
  9. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning11 May

Other references · 49

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  6. 2020Denoising Diffusion Implicit Modelscited by 9,781
  7. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  8. 2023Visual Instruction Tuningcited by 7,869
  9. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
  10. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  11. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,101
  12. 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,761
37 more
  1. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,253
  2. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  3. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,656
  4. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
  5. 2022DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Stepscited by 1,979
  6. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,919
  7. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,729
  8. 2019Towards VQA Models That Can Readcited by 1,718
  9. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,295
  10. 2021Cascaded Diffusion Models for High Fidelity Image Generationcited by 1,163
  11. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,121
  12. 2024Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Predictioncited by 1,023
  13. 2022Building Normalizing Flows with Stochastic Interpolantscited by 995
  14. 2024Chameleon: Mixed-Modal Early-Fusion Foundation Modelscited by 927
  15. 2015Generation and Comprehension of Unambiguous Object Descriptionscited by 904
  16. 2024Emu3: Next-Token Prediction is All You Needcited by 781
  17. 2024Autoregressive Model Beats Diffusion: Llama for Scalable Image Generationcited by 777
  18. 2024Show-o: One Single Transformer to Unify Multimodal Understanding and Generationcited by 752
  19. 2022Autoregressive Image Generation using Residual Quantizationcited by 722
  20. 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 710
  21. 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decodingcited by 688
  22. 2023MiniGPT-v2: large language model as a unified interface for vision-language multi-task learningcited by 625
  23. 2020Scaling Laws for Autoregressive Generative Modelingcited by 522
  24. 2020TextCaps: a Dataset for Image Captioning with Reading Comprehensioncited by 400
  25. 2022Analytic-DPM: an Analytic Estimate of the Optimal Reverse Variance in Diffusion Probabilistic Modelscited by 334
  26. 2024SEED-X: Multimodal Models with Unified Multi-granularity Comprehension and Generationcited by 331
  27. 2024Infinity: Scaling Bitwise AutoRegressive Modeling for High-Resolution Image Synthesiscited by 304
  28. 2023Woodpecker: Hallucination Correction for Multimodal Large Language Modelscited by 226
  29. 2023Any-to-Any Generation via Composable Diffusioncited by 204
  30. 2023Making LLaMA SEE and Draw with SEED Tokenizercited by 185
  31. 2024HALC: Object Hallucination Reduction via Adaptive Focal-Contrast Decodingcited by 174
  32. 2024Fluid: Scaling Autoregressive Text-to-image Generative Models with Continuous Tokenscited by 147
  33. 2024HART: Efficient Visual Generation with Hybrid Autoregressive Transformercited by 132
  34. 2024Liquid: Language Models are Scalable and Unified Multi-modal Generatorscited by 63
  35. 2023Unified Multi-Modal Latent Diffusion for Joint Subject and Text Conditional Image Generationcited by 60
  36. 2024X-VILA: Cross-Modality Alignment for Large Language Modelcited by 46
  37. 2024VAR-CLIP: Text-to-Image Generator with Visual Auto-Regressive Modelingcited by 42

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 29; 2026: 15.

44 papers.

Sort:

By area:

  1. 2025MMaDA: Multimodal Large Diffusion Language Modelscited by 250
  2. 2025Show-o2: Improved Native Unified Multimodal Modelscited by 250
  3. 2025Unified Multimodal Understanding and Generation Models: Advances, Challenges, and Opportunitiescited by 85
  4. 2025OneCAT: Decoder-Only Auto-Regressive Model for Unified Understanding and Generationcited by 50
  5. 2025Generative Multimodal Pretraining with Discrete Diffusion Timestep Tokenscited by 28
  6. 2025Lumos-1: On Autoregressive Video Generation with Discrete Diffusion from a Unified Model Perspectivecited by 22
  7. 2025VARGPT-v1.1: Improve Visual Autoregressive Large Unified Model via Iterative Instruction Tuning and Reinforcement Learningcited by 21
  8. 2025ComplexBench-Edit: Benchmarking Complex Instruction-Driven Image Editing via Compositional Dependenciescited by 21
  9. 2025SCALAR: Scale-wise Controllable Visual Autoregressive Learningcited by 18
  10. 2025GIR-Bench: Versatile Benchmark for Generating Images with Reasoningcited by 15
  11. 2026Demystifying Video Reasoningcited by 14
  12. 2026PackCache: A Training-Free Acceleration Method for Unified Autoregressive Video Generation via Compact KV-Cachecited by 13
32 more
  1. 2025Do we really have to filter out random noise in pre-training data for language models?cited by 11
  2. 2025iSegMan: Interactive Segment-and-Manipulate 3D Gaussianscited by 11
  3. 2025HBridge: H-Shape Bridging of Heterogeneous Experts for Unified Multimodal Understanding and Generationcited by 11
  4. 2025Semantic Context Matters: Improving Conditioning for Autoregressive Modelscited by 7
  5. 2026Ask, Solve, Generate: Self-Evolving Unified Multimodal Understanding and Generation via Self-Consistency Rewardscited by 7
  6. 2025GGBench: A Geometric Generative Reasoning Benchmark for Unified Multimodal Modelscited by 6
  7. 2025Instella-T2I: Pushing the Limits of 1D Discrete Latent Space Image Generationcited by 5
  8. 2025ARGenSeg: Image Segmentation with Autoregressive Image Generation Modelcited by 5
  9. 2025SAMPO:Scale-wise Autoregression with Motion PrOmpt for generative world modelscited by 4
  10. 2025A High-Quality Dataset and Reliable Evaluation for Interleaved Image-Text Generationcited by 3
  11. 2025SupCLAP: Controlling Optimization Trajectory Drift in Audio-Text Contrastive Learning with Support Vector Regularizationcited by 3
  12. 2025UniModel: A Visual-Only Framework for Unified Multimodal Understanding and Generationcited by 3
  13. 2026From Static Inference to Dynamic Interaction: A Survey of Streaming Large Language Modelscited by 3
  14. 2025LISA: A Layer-wise Integration and Suppression Approach for Hallucination Mitigation in Multimodal Large Language Modelscited by 2
  15. 2025ScaleWeaver: Weaving Efficient Controllable T2I Generation with Multi-Scale Reference Attentioncited by 2
  16. 2025WeMMU: Enhanced Bridging of Vision-Language Models and Diffusion Models via Noisy Query Tokenscited by 1
  17. 2025ViewMask-1-to-3: Multi-View Consistent Image Generation via Multimodal Discrete Diffusion Modelscited by 1
  18. 2026UniCompress: Token Compression for Unified Vision-Language Understanding and Generationcited by 1
  19. 2026WinTok: A Win-Win Hybrid Tokenizer via Decomposing Visual Understanding and Generation with Transferable Tokenscited by 1
  20. 2026HACK++: Towards More Effective Head-Aware Key-Value Compression for Efficient Visual Autoregressive Modelingcited by 1
  21. 2025Open Set Domain Adaptation with Vision-language models via Gradient-aware Separationcited by 0
  22. 2025EVLP:Learning Unified Embodied Vision-Language Planner with Reinforced Supervised Fine-Tuningcited by 0
  23. 2025DiverseVAR: Balancing Diversity and Quality of Next-Scale Visual Autoregressive Modelscited by 0
  24. 2025Unison: A Fully Automatic, Task-Universal, and Low-Cost Framework for Unified Understanding and Generationcited by 0
  25. 2026PnP-U3D: Plug-and-Play 3D Framework Bridging Autoregression and Diffusion for Unified Understanding and Generationcited by 0
  26. 2026StepVAR: Structure-Texture Guided Pruning for Visual Autoregressive Modelscited by 0
  27. 2026RAC: Rectified Flow Auto Codercited by 0
  28. 2026Semantic Generative Tuning for Unified Multimodal Modelscited by 0
  29. 2026Noisy-Channel Minimum Bayes Risk Decodingcited by 0
  30. 2026UniGen-AR: Unifying Visual Generation with Auto-Regressive Modelingcited by 0
  31. 2026Logit Refiner: Improving Visual Autoregressive Models via Intra-Scale Dependency Modelingcited by 0
  32. 2026Does Gradient Conflict Predict the Understanding--Generation Trade-off? A Controlled Audit of Conflict-Metric Validity in Unified Multimodal Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Growing Visual Generative Capacity for Pre-Trained MLLMs
  2. 2024The Revolution of Multimodal Large Language Models: A Survey
  3. 2026Predictive Regularization Against Visual Representation Degradation in Multimodal Large Language Models
  4. 2025Where MLLMs Attend and What They Rely On: Explaining Autoregressive Token Generation
  5. 2025Autoregressive Semantic Visual Reconstruction Helps VLMs Understand Better
  6. 2026UVU: Improving Multimodal Understanding via Vision-Language Unified Autoregressive Paradigm
  7. 2025Visual Representation Alignment for Multimodal Large Language Models
  8. 2025Perceiving Beyond Language Priors: Enhancing Visual Comprehension and Attention in Multimodal Models