mlsanityData to 13 Oct 2026

Any-to-Any Generation via Composable Diffusion

arXiv 2305.11846 ↗First posted 19 May 2023cs.CV cs.CL cs.LGcited by 203

Not on the map. Why?

Cited by 58 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We present Composable Diffusion (CoDi), a novel generative model capable of generating any combination of output modalities, such as language, image, video, or audio, from any combination of input modalities. Unlike existing generative AI systems, CoDi can generate multiple modalities in parallel and its input is not limited to a subset of modalities like text or image. Despite the absence of training datasets for many combinations of modalities, we propose to align modalities in both the input and output space. This allows CoDi to freely condition on any input combination and generate any group of modalities, even if they are not present in the training data. CoDi employs a novel composable generation strategy which involves building a shared multimodal space by bridging alignment in the diffusion process, enabling the synchronized generation of intertwined modalities, such as temporally aligned video and audio. Highly customizable and flexible, CoDi achieves strong joint-modality generation quality, and outperforms or is on par with the unimodal state-of-the-art for single-modality synthesis. The project page with demonstrations and code is at https://codi-gen.github.io

Key facts. Cited by 203 papers in the map (data through 13 Oct 2026). Builds on 51 of its 59 references that are in the map. First posted on arXiv 19 May 2023.

Builds on

51 of 59 references.

References · 51

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,950
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,304
  3. 2014Adam: A Method for Stochastic Optimizationcited by 37,093
  4. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,588
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,043
  6. 2020Denoising Diffusion Probabilistic Modelscited by 20,952
  7. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,777
  8. 2014Microsoft COCO: Common Objects in Contextcited by 17,550
  9. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 9,028
  10. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,023
  11. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,564
  12. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,103
38 more
  1. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,363
  2. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,141
  3. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,605
  4. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,586
  5. 2022Video Diffusion Modelscited by 2,293
  6. 2022Imagen Video: High Definition Video Generation with Diffusion Modelscited by 1,768
  7. 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,732
  8. 2023Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Modelscited by 1,467
  9. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,214
  10. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,176
  11. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,148
  12. 2022CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformerscited by 975
  13. 2021CogView: Mastering Text-to-Image Generation via Transformerscited by 692
  14. 2023Automatic Prompt Optimization with "Gradient Descent" and Beam Searchcited by 637
  15. 2023AudioLDM: Text-to-Audio Generation with Latent Diffusion Modelscited by 596
  16. 2023Structure and Content-Guided Video Synthesis with Diffusion Modelscited by 589
  17. 2021ClipCap: CLIP Prefix for Image Captioningcited by 558
  18. 2022GIT: A Generative Image-to-text Transformer for Vision and Languagecited by 536
  19. 2022Make-A-Scene: Scene-Based Text-to-Image Generation with Human Priorscited by 487
  20. 2021Unifying Vision-and-Language Tasks via Text Generationcited by 411
  21. 2023Make-An-Audio: Text-To-Audio Generation with Prompt-Enhanced Diffusion Modelscited by 324
  22. 2021MERLOT: Multimodal Neural Script Knowledge Modelscited by 311
  23. 2021GODIVA: Generating Open-DomaIn Videos from nAtural Descriptionscited by 255
  24. 2021N"UWA: Visual Synthesis Pre-training for Neural visUal World creAtioncited by 253
  25. 2016SoundNet: Learning Sound Representations from Unlabeled Videocited by 244
  26. 2021Advancing High-Resolution Video-Language Representation with Large-Scale Video Transcriptionscited by 226
  27. 2022Versatile Diffusion: Text, Images and Variations All in One Diffusion Modelcited by 199
  28. 2022MERLOT Reserve: Neural Script Knowledge through Vision and Language and Soundcited by 184
  29. 2023mPLUG-2: A Modularized Multi-modal Foundation Model Across Text, Image and Videocited by 154
  30. 2023VALOR: Vision-Audio-Language Omni-Perception Pretraining Model and Datasetcited by 138
  31. 2023Latent-Shift: Latent Diffusion with Temporal Shift for Efficient Text-to-Video Generationcited by 137
  32. 2022End-to-end Generative Pretraining for Multimodal Video Captioningcited by 116
  33. 2020Optimus: Organizing Sentences via Pre-trained Modeling of a Latent Spacecited by 114
  34. 2020Object Relational Graph with Teacher-Recommended Learning for Video Captioningcited by 107
  35. 2021ACAV100M: Automatic Curation of Large-Scale Datasets for Audio-Visual Video Representation Learningcited by 50
  36. 2022Semantic-Conditional Diffusion Networks for Image Captioningcited by 38
  37. 2022i-Code: An Integrative and Composable Multimodal Learning Frameworkcited by 24
  38. 2022TVLT: Textless Vision-Language Transformercited by 22

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2023: 31; 2024: 84; 2025: 70; 2026: 18.

203 papers.

Sort:

By area:

  1. 2024Show-o: One Single Transformer to Unify Multimodal Understanding and Generationcited by 740
  2. 2023NExT-GPT: Any-to-Any Multimodal LLMcited by 613
  3. 2023VideoPoet: A Large Language Model for Zero-Shot Video Generationcited by 482
  4. 2023DynamiCrafter: Animating Open-domain Images with Video Diffusion Priorscited by 466
  5. 2023Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Actioncited by 302
  6. 2024MM-LLMs: Recent Advances in MultiModal Large Language Modelscited by 278
  7. 2023Emu Video: Factorizing Text-to-Video Generation by Explicit Image Conditioningcited by 254
  8. 2025Show-o2: Improved Native Unified Multimodal Modelscited by 245
  9. 2025MMaDA: Multimodal Large Diffusion Language Modelscited by 244
  10. 2024AnyGPT: Unified Multimodal LLM with Discrete Sequence Modelingcited by 216
  11. 2023A Survey on Video Diffusion Modelscited by 206
  12. 2024MMAudio: Taming Multimodal Joint Training for High-Quality Video-to-Audio Synthesiscited by 134
38 more
  1. 2024CamI2V: Camera-Controlled Image-to-Video Diffusion Modelcited by 117
  2. 2024Seeing and Hearing: Open-domain Visual-Audio Generation with Diffusion Latent Alignerscited by 111
  3. 2024FoleyCrafter: Bring Silent Videos to Life with Lifelike and Synchronized Soundscited by 109
  4. 2023KOSMOS-2.5: A Multimodal Literate Modelcited by 95
  5. 2024Deep Multimodal Learning with Missing Modality: A Surveycited by 91
  6. 2024A Survey of Resource-efficient LLM and Multimodal Foundation Modelscited by 86
  7. 2023InstructVideo: Instructing Video Diffusion Models with Human Feedbackcited by 85
  8. 2024Pandora: Towards General World Model with Natural Language Actions and Video Statescited by 84
  9. 2025LaViDa: A Large Diffusion Language Model for Multimodal Understandingcited by 82
  10. 2024Dual Diffusion for Unified Image Generation and Understandingcited by 76
  11. 2024Exploring the Frontier of Vision-Language Models: A Survey of Current Methodologies and Future Directionscited by 68
  12. 2024MJ-Bench: Is Your Multimodal Reward Model Really a Good Judge for Text-to-Image Generation?cited by 64
  13. 2023CoDi-2: In-Context, Interleaved, and Interactive Any-to-Any Generationcited by 63
  14. 2023On the Design Fundamentals of Diffusion Models: A Surveycited by 60
  15. 2024A Comprehensive Review of Multimodal Large Language Models: Performance and Challenges Across Different Taskscited by 57
  16. 2024CuMo: Scaling Multimodal LLM with Co-Upcycled Mixture-of-Expertscited by 56
  17. 2025RealCam-I2V: Real-World Image-to-Video Generation with Interactive Complex Camera Controlcited by 55
  18. 2025OmniVinci: Enhancing Architecture and Data for Omni-Modal Understanding LLMcited by 55
  19. 2024Sora as a World Model? A Complete Survey on Text-to-Video Generationcited by 53
  20. 2024Vitron: A Unified Pixel-level Vision LLM for Understanding, Generating, Segmenting, Editingcited by 49
  21. 2023Breathing Life Into Sketches Using Text-to-Video Priorscited by 48
  22. 2023Smooth Diffusion: Crafting Smooth Latent Spaces in Diffusion Modelscited by 48
  23. 2024Read, Watch and Scream! Sound Generation from Text and Videocited by 48
  24. 2025Unified Multimodal Discrete Diffusioncited by 48
  25. 2024Multimodal Latent Language Modeling with Next-Token Diffusioncited by 47
  26. 2024X-VILA: Cross-Modality Alignment for Large Language Modelcited by 46
  27. 2024OmniFlow: Any-to-Any Generation with Multi-Modal Rectified Flowscited by 46
  28. 2024Diffusion Model-Based Video Editing: A Surveycited by 44
  29. 2025VARGPT: Unified Understanding and Generation in a Visual Autoregressive Multimodal Large Language Modelcited by 44
  30. 2025Parameter-Efficient Fine-Tuning for Foundation Modelscited by 44
  31. 2025Boosting Generative Image Modeling via Joint Image-Feature Synthesiscited by 43
  32. 2024WorldGPT: Empowering LLM as Multimodal World Modelcited by 42
  33. 2023STARSS23: An Audio-Visual Dataset of Spatial Recordings of Real Scenes with Spatiotemporal Annotations of Sound Eventscited by 40
  34. 2023Large AI Model Empowered Multimodal Semantic Communicationscited by 38
  35. 2024OmniBind: Large-scale Omni Multimodal Representation via Binding Spacescited by 38
  36. 2023GPT4Video: A Unified Multimodal Large Language Model for lnstruction-Followed Understanding and Safety-Aware Generationcited by 36
  37. 2025UniAVGen: Unified Audio and Video Generation with Asymmetric Cross-Modal Interactionscited by 36
  38. 2024Adversarial Attacks and Defenses on Text-to-Image Diffusion Models: A Surveycited by 34

50 most cited of 203 All 203 (JSON)