mlsanityData to 7 Oct 2026

MiniGPT-5: Interleaved Vision-and-Language Generation via Generative Vokens

arXiv 2310.02239 ↗First posted 3 Oct 2023cs.CV cs.AIcited by 15

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

The effectiveness of Multimodal Large Language Models (MLLMs) demonstrates a profound capability in multimodal understanding. However, the simultaneous generation of images with coherent texts is still underdeveloped. Addressing this, we introduce a novel interleaved vision-and-language generation method, centered around the concept of ``generative vokens". These vokens serve as pivotal elements contributing to coherent image-text outputs. Our method is marked by a unique two-stage training strategy for description-free multimodal generation, which does not necessitate extensive descriptions of images. We integrate classifier-free guidance to enhance the alignment of generated images and texts, ensuring more seamless and contextually relevant multimodal interactions. Our model, MiniGPT-5, exhibits substantial improvement over the baseline models on multimodal generation datasets, including MMDialog and VIST. The human evaluation shows MiniGPT-5 is better than the baseline model on more than 56% cases for multimodal generation, highlighting its efficacy across diverse benchmarks.

Key facts. Cited by 15 papers in the map (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 37 of its 58 references that are in the map. First posted on arXiv 3 Oct 2023.

Builds on

37 of 58 references.

Closest ancestors · 5

Same direction, newest first

  1. 2023DreamLLM: Synergistic Multimodal Comprehension and Creation20 Sep
  2. 2023Scaling Autoregressive Multi-Modal Models: Pretraining and Instruction Tuning5 Sep
  3. 2023Generating Images with Multimodal Language Models26 May
  4. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning11 May
  5. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuning5 May

Other references · 32

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  6. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  7. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  8. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  9. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  10. 2022Classifier-Free Diffusion Guidancecited by 5,717
  11. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  12. 2021Zero-Shot Text-to-Image Generationcited by 4,276
20 more
  1. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  2. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  3. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,595
  4. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  5. 2016Improved Techniques for Training GANscited by 2,938
  6. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  7. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,287
  8. 2023EVA-CLIP: Improved Training Techniques for CLIP at Scalecited by 786
  9. 2016Generative Adversarial Text to Image Synthesiscited by 735
  10. 2023Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Modelscited by 659
  11. 2021Multimodal Few-Shot Learning with Frozen Language Modelscited by 657
  12. 2023Muse: Text-To-Image Generation via Masked Generative Transformerscited by 640
  13. 2021Tip-Adapter: Training-free CLIP-Adapter for Better Vision-Language Modelingcited by 429
  14. 2016Visual Storytellingcited by 176
  15. 2023Planting a SEED of Vision in Large Language Modelcited by 140
  16. 2020Vokenization: Improving Language Understanding with Contextualized, Visual-Grounded Supervisioncited by 93
  17. 2022MMDialog: A Large-scale Multi-turn Dialogue Dataset Towards Multi-modal Open-domain Conversationcited by 56
  18. 2023Photoswap: Personalized Subject Swapping in Imagescited by 47
  19. 2021Multimodal Dialogue Response Generationcited by 32
  20. 2023Jointly Training Large Autoregressive Multimodal Modelscited by 32

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 5; 2025: 6; 2026: 2.

15 papers.

Sort:

By area:

  1. 2025Thinking with Images for Multimodal Reasoning: Foundations, Methods, and Future Frontierscited by 193
  2. 2024An Introduction to Vision-Language Modelingcited by 175
  3. 2024The Revolution of Multimodal Large Language Models: A Surveycited by 134
  4. 2023CapsFusion: Rethinking Image-Text Data at Scalecited by 90
  5. 2025A Survey of Multimodal Retrieval-Augmented Generationcited by 37
  6. 2024LLMs Meet Multimodal Generation and Editing: A Surveycited by 35
  7. 2024How Far Are We From AGI: Are LLMs All We Need?cited by 31
  8. 2025ERA: Transforming VLMs into Embodied Agents via Embodied Prior Learning and Online Reinforcement Learningcited by 10
  9. 2025Provoking Multi-modal Few-Shot LVLM via Exploration-Exploitation In-Context Learningcited by 5
  10. 2024From Linguistic Giants to Sensory Maestros: A Survey on Cross-Modal Reasoning with Large Language Modelscited by 4
  11. 2025Omni-AutoThink: Adaptive Multimodal Reasoning via Reinforcement Learningcited by 4
  12. 2023How to Bridge the Gap between Modalities: Survey on Multimodal Large Language Modelcited by 3
3 more
  1. 2025Bridging the Gap Between Multimodal Foundation Models and World Modelscited by 1
  2. 2026ViSTR-Bench: Can MLLMs Reason from Continuous Visual Cues in Dynamic Scenes?cited by 1
  3. 2026Self-correction Optimization for Interleaved Multimodal Generationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024OpenING: A Comprehensive Benchmark for Judging Open-ended Interleaved Image-Text Generation
  2. 2026Towards Unified Multimodal Interleaved Generation via Group Relative Policy Optimization
  3. 2023VLIS: Unimodal Language Models Guide Multimodal Language Generation
  4. 2026DuoGen: Towards General Purpose Interleaved Multimodal Generation
  5. 2024The Narrow Gate: Localized Image-Text Communication in Native Multimodal Models
  6. 2024A Comprehensive Survey and Guide to Multimodal Large Language Models in Vision-Language Tasks
  7. 2026Let ViT Speak: Generative Language-Image Pre-training
  8. 2025ARMOR: Empowering Multimodal Understanding Model with Interleaved Multimodal Generation Capability