mlsanityData to 14 Oct 2026

Visual Generation Tuning

arXiv 2511.23469 ↗First posted 28 Nov 2025cs.CVcited by 2

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Large Vision Language Models (VLMs) effectively bridge the modality gap through extensive pretraining, acquiring sophisticated visual representations aligned with language. However, it remains underexplored whether these representations, optimized for multimodal understanding tasks, harbor an inherent potential for visual generation. In this paper, we propose VGT, Visual Generation Tuning, a novel paradigm designed to stimulate the underlying capabilities of visual generation within any vision language models. By performing efficient visual generation tuning on well-pretrained VLMs, we significantly mitigate the alignment costs and accelerate the convergence of autoregressive modeling in the continuous space (20x speedup). Specifically, we dismiss the entangled pixel-level VAEs designed for diffusion transformers and formulate VGT-AE through aligning the semantic encoders from pretrained VLMs with the latent representations of pixel decoders. In image reconstruction tasks, we achieve 26.67 PSNR and 0.50 rFID at a 28x compression ratio, outperforming specialized VAEs; in visual generation tasks, we achieve state-of-the-art outcomes among autoregressive models, 0.77 on GenEval and 78.73 on DPG-Bench. Furthermore, our proposed VGT showcases significant scaling promise and is versatile for endowing any VLMs trained for multimodal understanding with the capabilities of visual generation, which paves the new avenue to explore next-generation unified multimodal foundation models. Models and codes are available at https://github.com/hustvl/VGT.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 61 of its 69 references that are in the map. First posted on arXiv 28 Nov 2025.

Builds on

61 of 69 references.

References · 61

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  3. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,177
  4. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  5. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
  6. 2025Qwen2.5-VL Technical Reportcited by 5,587
  7. 2017Neural Discrete Representation Learningcited by 4,712
  8. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,356
  9. 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesiscited by 4,181
  10. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,253
  11. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,141
  12. 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,834
38 more
  1. 2019Towards VQA Models That Can Readcited by 1,718
  2. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,652
  3. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,216
  4. 2022MaskGIT: Masked Generative Image Transformercited by 1,099
  5. 2024Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Predictioncited by 1,023
  6. 2023GenEval: An Object-Focused Framework for Evaluating Text-to-Image Alignmentcited by 972
  7. 2024Chameleon: Mixed-Modal Early-Fusion Foundation Modelscited by 927
  8. 2023PixArt-α: Fast Training of Diffusion Transformer for Photorealistic Text-to-Image Synthesiscited by 864
  9. 2025Janus-Pro: Unified Multimodal Understanding and Generation with Data and Model Scalingcited by 812
  10. 2016A Diagram Is Worth A Dozen Imagescited by 804
  11. 2024Emu3: Next-Token Prediction is All You Needcited by 781
  12. 2024Autoregressive Model Beats Diffusion: Llama for Scalable Image Generationcited by 777
  13. 2024Representation Alignment for Generation: Training Diffusion Transformers Is Easier Than You Thinkcited by 703
  14. 2024Autoregressive Image Generation without Vector Quantizationcited by 651
  15. 2024ELLA: Equip Diffusion Models with LLM for Enhanced Semantic Alignmentcited by 475
  16. 2024Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and Generationcited by 457
  17. 2024OmniGen: Unified Image Generationcited by 394
  18. 2025Reconstruction vs. Generation: Taming Optimization Dilemma in Latent Diffusion Modelscited by 393
  19. 2025BLIP3-o: A Family of Fully Open Unified Multimodal Models-Architecture, Training and Datasetcited by 376
  20. 2024SEED-X: Multimodal Models with Unified Multi-granularity Comprehension and Generationcited by 331
  21. 2022BEiT v2: Masked Image Modeling with Vector-Quantized Visual Tokenizerscited by 303
  22. 2024VILA-U: a Unified Foundation Model Integrating Visual Understanding and Generationcited by 295
  23. 2025Diffusion Transformers with Representation Autoencoderscited by 295
  24. 2025REPA-E: Unlocking VAE for End-to-End Tuning with Latent Diffusion Transformerscited by 225
  25. 2024Deep Compression Autoencoder for Efficient High-Resolution Diffusion Modelscited by 217
  26. 2023JourneyDB: A Benchmark for Generative Image Understandingcited by 205
  27. 2024TokenFlow: Unified Image Tokenizer for Multimodal Understanding and Generationcited by 202
  28. 2024Open-MAGVIT2: An Open-Source Project Toward Democratizing Auto-regressive Visual Generationcited by 155
  29. 2025UniTok: A Unified Tokenizer for Visual Generation and Understandingcited by 153
  30. 2024Fluid: Scaling Autoregressive Text-to-image Generative Models with Continuous Tokenscited by 147
  31. 2024Randomized Autoregressive Visual Generationcited by 127
  32. 2025X-Omni: Reinforcement Learning Makes Discrete Autoregressive Image Generative Models Great Againcited by 112
  33. 2025Latent Diffusion Model without Variational Autoencodercited by 88
  34. 2025SimpleAR: Pushing the Frontier of Autoregressive Visual Generation through Pretraining, SFT, and RLcited by 86
  35. 2024RandAR: Decoder-only Autoregressive Visual Generation in Random Orderscited by 83
  36. 2023GIVT: Generative Infinite-Vocabulary Transformerscited by 80
  37. 2024Addressing Representation Collapse in Vector Quantized Models with One Linear Layercited by 80
  38. 2024FasterDiT: Towards Faster Diffusion Transformers Training without Architecture Modificationcited by 64

50 most cited of 61 All 61 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Can Unified Generation and Understanding Models Maintain Semantic Equivalence Across Different Output Modalities?cited by 1
  2. 2026Multimodal Flow: Unified Flow Modeling of Language and Vision in Embedding Spacescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025VUGEN: Visual Understanding priors for GENeration
  2. 2026UniGen-AR: Unifying Visual Generation with Auto-Regressive Modeling
  3. 2026High-Fidelity Text-to-Image Generation from Pre-Trained Vision-Language Models via Distribution-Conditioned Diffusion Decoding
  4. 2025Cross-Task Generalization Between Understanding and Generation in Unified Vision-Language Models: A Controlled Study
  5. 2025Growing Visual Generative Capacity for Pre-Trained MLLMs
  6. 2025VARGPT: Unified Understanding and Generation in a Visual Autoregressive Multimodal Large Language Model
  7. 2026Generation as Auxiliary Supervision: Enhancing Visual Understanding at Zero Inference Overhead via Decoupled Embedding Prediction
  8. 2024MUSE-VL: Modeling Unified VLM through Semantic Discrete Encoding