mlsanityData to 14 Oct 2026

CoDi-2: In-Context, Interleaved, and Interactive Any-to-Any Generation

arXiv 2311.18775 ↗First posted 30 Nov 2023cs.CV cs.AI cs.CLcited by 63

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

We present CoDi-2, a versatile and interactive Multimodal Large Language Model (MLLM) that can follow complex multimodal interleaved instructions, conduct in-context learning (ICL), reason, chat, edit, etc., in an any-to-any input-output modality paradigm. By aligning modalities with language for both encoding and generation, CoDi-2 empowers Large Language Models (LLMs) to not only understand complex modality-interleaved instructions and in-context examples, but also autoregressively generate grounded and coherent multimodal outputs in the continuous feature space. To train CoDi-2, we build a large-scale generation dataset encompassing in-context multimodal instructions across text, vision, and audio. CoDi-2 demonstrates a wide range of zero-shot capabilities for multimodal generation, such as in-context learning, reasoning, and compositionality of any-to-any modality generation through multi-round interactive conversation. CoDi-2 surpasses previous domain-specific models on tasks such as subject-driven image generation, vision transformation, and audio editing. CoDi-2 signifies a substantial breakthrough in developing a comprehensive multimodal foundation model adept at interpreting in-context language-vision-audio interleaved instructions and producing multimodal outputs.

Key facts. Cited by 63 papers in the map (data through 14 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 40 of its 48 references that are in the map. First posted on arXiv 30 Nov 2023.

Builds on

40 of 48 references.

Closest ancestors · 4

Same direction, newest first

  1. 2023Kosmos-G: Generating Images in Context with Multimodal Large Language Models4 Oct
  2. 2023NExT-GPT: Any-to-Any Multimodal LLM11 Sep
  3. 2023Generating Images with Multimodal Language Models26 May
  4. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuning5 May

Other references · 36

  1. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  5. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,255
  6. 2023Visual Instruction Tuningcited by 7,869
  7. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
  8. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,503
  9. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,300
  10. 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,611
  11. 2023A Survey of Large Language Modelscited by 2,576
  12. 2021SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equationscited by 1,937
24 more
  1. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,555
  2. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,177
  3. 2023ImageBind: One Embedding Space To Bind Them Allcited by 1,175
  4. 2022Multi-Concept Customization of Text-to-Image Diffusioncited by 1,106
  5. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,040
  6. 2021CogView: Mastering Text-to-Image Generation via Transformerscited by 694
  7. 2023LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Modelcited by 602
  8. 2022GIT: A Generative Image-to-text Transformer for Vision and Languagecited by 537
  9. 2023Ferret: Refer and Ground Anything Anywhere at Any Granularitycited by 509
  10. 2023BLIP-Diffusion: Pre-trained Subject Representation for Controllable Text-to-Image Generation and Editingcited by 442
  11. 2023Chameleon: Plug-and-Play Compositional Reasoning with Large Language Modelscited by 356
  12. 2023AudioLDM 2: Learning Holistic Audio Generation with Self-supervised Pretrainingcited by 344
  13. 2022Diffsound: Discrete Diffusion Model for Text-to-sound Generationcited by 257
  14. 2022Re-Imagen: Retrieval-Augmented Text-to-Image Generatorcited by 210
  15. 2023Any-to-Any Generation via Composable Diffusioncited by 204
  16. 2022MERLOT Reserve: Neural Script Knowledge through Vision and Language and Soundcited by 184
  17. 2023VALOR: Vision-Audio-Language Omni-Perception Pretraining Model and Datasetcited by 140
  18. 2022End-to-end Generative Pretraining for Multimodal Video Captioningcited by 116
  19. 2020Object Relational Graph with Teacher-Recommended Learning for Video Captioningcited by 107
  20. 2023In-Context Learning Unlocked for Diffusion Modelscited by 104
  21. 2023The Generative AI Paradox: "What It Can Create, It May Not Understand"cited by 98
  22. 2023AUDIT: Audio Editing by Following Instructions with Latent Diffusion Modelscited by 83
  23. 2023ReVersion: Diffusion-Based Relation Inversion from Imagescited by 82
  24. 2022i-Code: An Integrative and Composable Multimodal Learning Frameworkcited by 24

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 30; 2025: 25; 2026: 8.

63 papers.

Sort:

By area:

  1. 2024OmniGen: Unified Image Generationcited by 394
  2. 2024MM-LLMs: Recent Advances in MultiModal Large Language Modelscited by 278
  3. 2025Show-o2: Improved Native Unified Multimodal Modelscited by 250
  4. 2024AnyGPT: Unified Multimodal LLM with Discrete Sequence Modelingcited by 217
  5. 2024Audio Flamingo: A Novel Audio Language Model with Few-Shot Learning and Dialogue Abilitiescited by 186
  6. 2024EarthGPT: A Universal Multi-modal Large Language Model for Multi-sensor Image Comprehension in Remote Sensing Domaincited by 184
  7. 2024The Revolution of Multimodal Large Language Models: A Surveycited by 137
  8. 2024MMAudio: Taming Multimodal Joint Training for High-Quality Video-to-Audio Synthesiscited by 134
  9. 2024Exploring the Frontier of Vision-Language Models: A Survey of Current Methodologies and Future Directionscited by 68
  10. 2024Sora as a World Model? A Complete Survey on Text-to-Video Generationcited by 53
  11. 2025Parameter-Efficient Fine-Tuning for Foundation Modelscited by 45
  12. 2025Visual Large Language Models for Generalized and Specialized Applicationscited by 36
38 more
  1. 2024LLMs Meet Multimodal Generation and Editing: A Surveycited by 35
  2. 2024OpenING: A Comprehensive Benchmark for Judging Open-ended Interleaved Image-Text Generationcited by 31
  3. 2024SPHERE: Unveiling Spatial Blind Spots in Vision-Language Models Through Hierarchical Evaluationcited by 31
  4. 2024Uncertainty-Aware Evaluation for Vision-Language Modelscited by 29
  5. 2024Taming Data and Transformers for Audio Generationcited by 28
  6. 2024Next Token Prediction Towards Multimodal Intelligence: A Comprehensive Surveycited by 27
  7. 2026JavisDiT++: Unified Modeling and Optimization for Joint Audio-Video Generationcited by 25
  8. 2024From Specific-MLLMs to Omni-MLLMs: A Survey on MLLMs Aligned with Multi-modalitiescited by 22
  9. 2024The Evolution of Multimodal Model Architecturescited by 21
  10. 2024An overview of domain-specific foundation model: key technologies, applications and challengescited by 19
  11. 2025Generative AI for Cel-Animation: A Surveycited by 17
  12. 2025A Survey of Foundation Model-Powered Recommender Systems: From Feature-Based, Generative to Agentic Paradigmscited by 17
  13. 2024VMAS: Video-to-Music Generation via Semantic Alignment in Web Music Videoscited by 15
  14. 2025Zero-Shot Audio-Visual Editing via Cross-Modal Delta Denoisingcited by 14
  15. 2024UnifiedMLLM: Enabling Unified Representation for Multi-modal Multi-tasks With Large Language Modelcited by 13
  16. 2024Unleashing In-context Learning of Autoregressive Models for Few-shot Image Manipulationcited by 13
  17. 2025X-Fusion: Introducing New Modality to Frozen Large Language Modelscited by 13
  18. 2025MAGNET: A Multi-agent Framework for Finding Audio-Visual Needles by Reasoning over Multi-Video Haystackscited by 12
  19. 2026UniM: A Unified Any-to-Any Interleaved Multimodal Benchmarkcited by 11
  20. 2024SEE-2-SOUND: Zero-Shot Spatial Environment-to-Spatial Soundcited by 10
  21. 2024Towards Unifying Understanding and Generation in the Era of Vision Foundation Models: A Survey from the Autoregression Perspectivecited by 10
  22. 2024Modality-Specialized Synergizers for Interleaved Vision-Language Generalistscited by 9
  23. 2024Multi-modal Generative AI: Multi-modal LLMs, Diffusions, and the Unificationcited by 8
  24. 2025KeyVID: Keyframe-Aware Video Diffusion for Audio-Synchronized Visual Animationcited by 8
  25. 2026Evolution of Video Generative Foundationscited by 7
  26. 2025GPS as a Control Signal for Image Generationcited by 5
  27. 2025DeepAudio-V1:Towards Multi-Modal Multi-Stage End-to-End Video to Speech and Audio Generationcited by 5
  28. 2025Uncertainty-o: One Model-agnostic Framework for Unveiling Uncertainty in Large Multimodal Modelscited by 5
  29. 2025Pisces: An Auto-regressive Foundation Model for Image Understanding and Generationcited by 5
  30. 2026Dynin-Omni: Omnimodal Unified Large Diffusion Language Modelcited by 5
  31. 2024Visual Echoes: A Simple Unified Transformer for Audio-Visual Generationcited by 4
  32. 2024Generative Visual Instruction Tuningcited by 4
  33. 2025EgoAdapt: Adaptive Multisensory Distillation and Policy Learning for Efficient Egocentric Perceptioncited by 4
  34. 2025AMUSE: Audio-Visual Benchmark and Alignment Framework for Agentic Multi-Speaker Understandingcited by 4
  35. 2024C3LLM: Conditional Multimodal Content Generation Using Large Language Modelscited by 3
  36. 2024Multimodal Contrastive In-Context Learningcited by 3
  37. 2025Sounding that Object: Interactive Object-Aware Image to Audio Generationcited by 3
  38. 2025AudioStory: Generating Long-Form Narrative Audio with Large Language Modelscited by 3

50 most cited of 63 All 63 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023ChatBridge: Bridging Modalities with Large Language Model as a Language Catalyst
  2. 2024A Comprehensive Survey and Guide to Multimodal Large Language Models in Vision-Language Tasks
  3. 2024A Comprehensive Review of Multimodal Large Language Models: Performance and Challenges Across Different Tasks
  4. 2023SEED-Bench-2: Benchmarking Multimodal Large Language Models
  5. 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Models
  6. 2023On the Performance of Multimodal Language Models
  7. 2026ChatUMM: Robust Context Tracking for Conversational Interleaved Generation
  8. 2025ARMOR: Empowering Multimodal Understanding Model with Interleaved Multimodal Generation Capability