mlsanityData to 14 Oct 2026

Kosmos-G: Generating Images in Context with Multimodal Large Language Models

arXiv 2310.02992 ↗First posted 4 Oct 2023cs.CV cs.CLcited by 105

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Recent advancements in subject-driven image generation have made significant strides. However, current methods still fall short in diverse application scenarios, as they require test-time tuning and cannot accept interleaved multi-image and text input. These limitations keep them far from the ultimate goal of "image as a foreign language in image generation." This paper presents Kosmos-G, a model that leverages the advanced multimodal perception capabilities of Multimodal Large Language Models (MLLMs) to tackle the aforementioned challenge. Our approach aligns the output space of MLLM with CLIP using the textual modality as an anchor and performs compositional instruction tuning on curated data. Kosmos-G demonstrates an impressive capability of zero-shot subject-driven generation with interleaved multi-image and text input. Notably, the score distillation instruction tuning requires no modifications to the image decoder. This allows for a seamless substitution of CLIP and effortless integration with a myriad of U-Net techniques ranging from fine-grained controls to personalized image decoder variants. We posit Kosmos-G as an initial attempt towards the goal of "image as a foreign language in image generation." The code can be found at https://aka.ms/Kosmos-G

Key facts. Cited by 105 papers in the map (data through 14 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 48 of its 54 references that are in the map. First posted on arXiv 4 Oct 2023.

Builds on

48 of 54 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023DreamLLM: Synergistic Multimodal Comprehension and Creation20 Sep
  2. 2023Generating Images with Multimodal Language Models26 May

Other references · 46

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  4. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,713
  5. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  6. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  7. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  8. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  9. 2020Denoising Diffusion Implicit Modelscited by 9,781
  10. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,058
  11. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
  12. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,355
34 more
  1. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  2. 2022Classifier-Free Diffusion Guidancecited by 5,847
  3. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,503
  4. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  5. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
  6. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,614
  7. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,300
  8. 2022DreamFusion: Text-to-3D using 2D Diffusioncited by 3,025
  9. 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,611
  10. 2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,245
  11. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,989
  12. 2022DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Stepscited by 1,979
  13. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,555
  14. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,216
  15. 2022Multi-Concept Customization of Text-to-Image Diffusioncited by 1,106
  16. 2023Language Is Not All You Need: Aligning Perception with Language Modelscited by 543
  17. 2021Image Segmentation Using Text and Image Promptscited by 512
  18. 2022Make-A-Scene: Scene-Based Text-to-Image Generation with Human Priorscited by 487
  19. 2023BLIP-Diffusion: Pre-trained Subject Representation for Controllable Text-to-Image Generation and Editingcited by 442
  20. 2023ELITE: Encoding Visual Concepts into Textual Embeddings for Customized Text-to-Image Generationcited by 416
  21. 2022Understanding Diffusion Models: A Unified Perspectivecited by 340
  22. 2023FastComposer: Tuning-Free Multi-Subject Image Generation with Localized Attentioncited by 313
  23. 2023Break-A-Scene: Extracting Multiple Concepts from a Single Imagecited by 231
  24. 2023Subject-driven Text-to-Image Generation via Apprenticeship Learningcited by 212
  25. 2022Re-Imagen: Retrieval-Augmented Text-to-Image Generatorcited by 210
  26. 2023Key-Locked Rank One Editing for Text-to-Image Personalizationcited by 194
  27. 2023Emu: Generative Pretraining in Multimodalitycited by 157
  28. 2022CM3: A Causal Masked Multimodal Model of the Internetcited by 142
  29. 2023Continual Diffusion: Continual Customization of Text-to-Image Diffusion with C-LoRAcited by 141
  30. 2023In-Context Learning Unlocked for Diffusion Modelscited by 104
  31. 2022Language Models are General-Purpose Interfacescited by 97
  32. 2023GlueGen: Plug and Play Multi-modal Encoders for X-to-image Generationcited by 25
  33. 2022TorchScale: Transformers at Scalecited by 10
  34. 2022M-VADER: A Model for Diffusion with Multimodal Contextcited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 6; 2024: 42; 2025: 45; 2026: 12.

105 papers.

Sort:

By area:

  1. 2023Generative Multimodal Models are In-Context Learnerscited by 446
  2. 2024OmniGen: Unified Image Generationcited by 394
  3. 2024OminiControl: Minimal and Universal Control for Diffusion Transformercited by 340
  4. 2023Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Actioncited by 302
  5. 2024UltraEdit: Instruction-based Fine-Grained Image Editing at Scalecited by 268
  6. 2025Transfer between Modalities with MetaQueriescited by 236
  7. 2024MetaMorph: Multimodal Understanding and Generation via Instruction Tuningcited by 206
  8. 2024MS-Diffusion: Multi-subject Zero-shot Image Personalization with Layout Guidancecited by 144
  9. 2024The Revolution of Multimodal Large Language Models: A Surveycited by 137
  10. 2024ShapeLLM: Universal 3D Object Understanding for Embodied Interactioncited by 137
  11. 2024VisionLLM v2: An End-to-End Generalist Multimodal Large Language Model for Hundreds of Vision-Language Taskscited by 137
  12. 2024DreamBench++: A Human-Aligned Benchmark for Personalized Image Generationcited by 131
38 more
  1. 2024MovieDreamer: Hierarchical Generation for Coherent Long Visual Sequencecited by 89
  2. 2025Unified Multimodal Understanding and Generation Models: Advances, Challenges, and Opportunitiescited by 85
  3. 2024Controllable Generation with Text-to-Image Diffusion Models: A Surveycited by 83
  4. 2024Perception Tokens Enhance Visual Reasoning in Multimodal Language Modelscited by 71
  5. 2024Instruct-Imagen: Image Generation with Multi-modal Instructioncited by 65
  6. 2023CoDi-2: In-Context, Interleaved, and Interactive Any-to-Any Generationcited by 63
  7. 2024λ-ECLIPSE: Multi-Concept Personalized Text-to-Image Diffusion Models by Leveraging CLIP Latent Spacecited by 51
  8. 2024Large-Scale Text-to-Image Model with Inpainting is a Zero-Shot Subject-Driven Image Generatorcited by 48
  9. 2023LLaVA-Interactive: An All-in-One Demo for Image Chat, Segmentation, Generation and Editingcited by 47
  10. 2025Latent Sketchpad: Sketching Visual Thoughts to Elicit Multimodal Reasoning in MLLMscited by 43
  11. 2023VL-GPT: A Generative Pre-trained Transformer for Vision and Language Understanding and Generationcited by 41
  12. 2025DesignDiffusion: High-Quality Text-to-Design Image Generation with Diffusion Modelscited by 40
  13. 2025DICEPTION: A Generalist Diffusion Model for Visual Perceptual Taskscited by 39
  14. 2025A Survey of Multimodal Retrieval-Augmented Generationcited by 37
  15. 2024FreeEdit: Mask-free Reference-based Image Editing with Multi-modal Instructioncited by 36
  16. 2025Visual Large Language Models for Generalized and Specialized Applicationscited by 36
  17. 2024DreamOmni: Unified Image Generation and Editingcited by 30
  18. 2024A Survey on Personalized Content Synthesis with Diffusion Modelscited by 29
  19. 2024Next Token Prediction Towards Multimodal Intelligence: A Comprehensive Surveycited by 27
  20. 2025Modality Curation: Building Universal Embeddings for Advanced Multimodal Information Retrievalcited by 26
  21. 2025Generating Multi-Image Synthetic Data for Text-to-Image Customizationcited by 25
  22. 2024Multi-Reward as Condition for Instruction-based Image Editingcited by 20
  23. 2024LoRA.rar: Learning to Merge LoRAs via Hypernetworks for Subject-Style Conditioned Image Generationcited by 19
  24. 2024PatchDPO: Patch-level DPO for Finetuning-free Personalized Image Generationcited by 18
  25. 2025Multimodal Representation Alignment for Image Generation: Text-Image Interleaved Control Is Easier Than You Thinkcited by 18
  26. 2025SuperEdit: Rectifying and Facilitating Supervision for Instruction-Based Image Editingcited by 18
  27. 2024Large Language Model Can Transcribe Speech in Multi-Talker Scenarios with Versatile Instructionscited by 17
  28. 2025A Survey of Foundation Model-Powered Recommender Systems: From Feature-Based, Generative to Agentic Paradigmscited by 17
  29. 2024RealCustom++: Representing Images as Real Textual Word for Real-Time Customizationcited by 16
  30. 2025Exploring the Deep Fusion of Large Language Models and Diffusion Transformers for Text-to-Image Synthesiscited by 16
  31. 2025VisCodex: Unified Multimodal Code Generation via Merging Vision and Coding Modelscited by 16
  32. 2024EMMA: Your Text-to-Image Diffusion Model Can Secretly Accept Multi-Modal Promptscited by 14
  33. 2024MotionBank: A Large-scale Video Motion Benchmark with Disentangled Rule-based Annotationscited by 14
  34. 2024CustomContrast: A Multilevel Contrastive Perspective For Subject-Driven Text-to-Image Customizationcited by 13
  35. 2024GroundingBooth: Grounding Text-to-Image Customizationcited by 13
  36. 2024Novel Object Synthesis via Adaptive Text-Image Harmonycited by 12
  37. 2025I Think, Therefore I Diffuse: Enabling Multimodal In-Context Reasoning in Diffusion Modelscited by 12
  38. 2023Customization Assistant for Text-to-image Generationcited by 11

50 most cited of 105 All 105 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025MM-R1: Unleashing the Power of Unified Multimodal Large Language Models for Personalized Image Generation
  2. 2023MiniGPT-5: Interleaved Vision-and-Language Generation via Generative Vokens
  3. 2024Divide, Conquer and Combine: A Training-Free Framework for High-Resolution Image Perception in Multimodal Large Language Models
  4. 2023On the Performance of Multimodal Language Models
  5. 2025Demystifying the Visual Quality Paradox in Multimodal Large Language Models
  6. 2025Boosting Text-To-Image Generation via Multilingual Prompting in Large Multimodal Models
  7. 2023StableLLaVA: Enhanced Visual Instruction Tuning with Synthesized Image-Dialogue Data
  8. 2024Generative Visual Instruction Tuning