mlsanityData to 14 Oct 2026

Generative Visual Instruction Tuning

arXiv 2406.11262 ↗First posted 17 Jun 2024cs.CVcited by 4

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

We propose to use automatically generated instruction-following data to improve the zero-shot capabilities of a large multimodal model with additional support for generative and image editing tasks. We achieve this by curating a new multimodal instruction-following set using GPT-4V and existing datasets for image generation and editing. Using this instruction set and the existing LLaVA-Finetune instruction set for visual understanding tasks, we produce GenLLaVA, a Generative Large Language and Visual Assistant. GenLLaVA is built through a strategy that combines three types of large pretrained models through instruction finetuning: Mistral for language modeling, SigLIP for image-text matching, and StableDiffusion for text-to-image generation. Our model demonstrates visual understanding capabilities superior to LLaVA and additionally demonstrates competitive results with native multimodal models such as Unified-IO 2, paving the way for building advanced general-purpose visual assistants by effectively re-using existing multimodal models. We open-source our dataset, codebase, and model checkpoints to foster further research and application in this domain.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 54 of its 64 references that are in the map. First posted on arXiv 17 Jun 2024.

Builds on

54 of 64 references.

Closest ancestors · 12

Same direction, newest first

  1. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suites25 Apr
  2. 2024AnyGPT: Unified Multimodal LLM with Discrete Sequence Modeling19 Feb
  3. 2023Generative Multimodal Models are In-Context Learners20 Dec
  4. 2023CoDi-2: In-Context, Interleaved, and Interactive Any-to-Any Generation30 Nov
  5. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captions21 Nov
  6. 2023Improved Baselines with Visual Instruction Tuning5 Oct
  7. 2023DreamLLM: Synergistic Multimodal Comprehension and Creation20 Sep
  8. 2023Scaling Autoregressive Multi-Modal Models: Pretraining and Instruction Tuning5 Sep
  9. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities4 Aug
  10. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehension30 Jul
  11. 2023MMBench: Is Your Multi-modal Model an All-around Player?12 Jul
  12. 2023Generating Images with Multimodal Language Models26 May

Other references · 42

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  5. 2023GPT-4 Technical Reportcited by 18,396
  6. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  7. 2023Visual Instruction Tuningcited by 7,869
  8. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  9. 2022Classifier-Free Diffusion Guidancecited by 5,847
  10. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
  11. 2023Qwen Technical Reportcited by 3,344
  12. 2023Mistral 7Bcited by 3,164
30 more
  1. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
  2. 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,611
  3. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,592
  4. 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,431
  5. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,080
  6. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,614
  7. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,479
  8. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,402
  9. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,317
  10. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,301
  11. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,216
  12. 2022MaskGIT: Masked Generative Image Transformercited by 1,099
  13. 2024Chameleon: Mixed-Modal Early-Fusion Foundation Modelscited by 927
  14. 2016A Diagram Is Worth A Dozen Imagescited by 804
  15. 2022Autoregressive Image Generation using Residual Quantizationcited by 722
  16. 2024Yi: Open Foundation Models by 01.AIcited by 719
  17. 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 708
  18. 2023Ferret: Refer and Ground Anything Anywhere at Any Granularitycited by 509
  19. 2023Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuningcited by 445
  20. 2021OCR-free Document Understanding Transformercited by 407
  21. 2018DVQA: Understanding Data Visualizations via Question Answeringcited by 340
  22. 2024Prismatic VLMs: Investigating the Design Space of Visually-Conditioned Language Modelscited by 339
  23. 2024What matters when building vision-language models?cited by 326
  24. 2023Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Actioncited by 302
  25. 2024LLaVA-UHD: an LMM Perceiving Any Aspect Ratio and High-Resolution Imagescited by 195
  26. 2023Guiding Instruction-based Image Editing via Multimodal Large Language Modelscited by 185
  27. 2023SVIT: Scaling up Visual Instruction Tuningcited by 162
  28. 2023Emu: Generative Pretraining in Multimodalitycited by 157
  29. 2023To See is to Believe: Prompting GPT-4V for Better Visual Instruction Tuningcited by 144
  30. 2019Expressing Visual Relationships via Languagecited by 49

Built on by

Cited by year

Citing papers per year: 2024: 3; 2025: 1.

4 papers.

  1. 2024Unleashing In-context Learning of Autoregressive Models for Few-shot Image Manipulationcited by 13
  2. 2024Towards Unifying Understanding and Generation in the Era of Vision Foundation Models: A Survey from the Autoregression Perspectivecited by 10
  3. 2024INF-LLaVA: Dual-perspective Perception for High-Resolution Multimodal Large Language Modelcited by 4
  4. 2025Diffusion Instruction Tuningcited by 2

Similar, not cited

Close in wording, no citation either way

  1. 2023StableLLaVA: Enhanced Visual Instruction Tuning with Synthesized Image-Dialogue Data
  2. 2024MetaMorph: Multimodal Understanding and Generation via Instruction Tuning
  3. 2025VARGPT-v1.1: Improve Visual Autoregressive Large Unified Model via Iterative Instruction Tuning and Reinforcement Learning
  4. 2024MLAN: Language-Based Instruction Tuning Preserves and Transfers Knowledge in Multimodal Language Models
  5. 2024A High-Quality Text-Rich Image Instruction Tuning Dataset via Hybrid Instruction Generation
  6. 2023Genixer: Empowering Multimodal Large Language Models as a Powerful Data Generator
  7. 2025MIGE: Mutually Enhanced Multimodal Instruction-Based Image Generation and Editing
  8. 2023Position-Enhanced Visual Instruction Tuning for Multimodal Large Language Models