# Draw Your Art Dream: Diverse Digital Art Synthesis with Multimodal Guided Diffusion

arXiv: https://arxiv.org/abs/2209.13360 · HTML: https://mlsanity.com/paper/2209.13360/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models.
- Builds on 53 of its 76 references that are in the map.
- First posted on arXiv 27 Sep 2022.

## Placed in

- Diffusion models / [Image generation, text-to-image and editing](https://mlsanity.com/diffusion/image-generation-text-to-image-and-editing/)

## Builds on (most cited first, 30 of 53)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018)
- [Classifier-Free Diffusion Guidance](https://mlsanity.com/paper/2207.12598/) (2022)
- [Generative Adversarial Networks](https://mlsanity.com/paper/2203.00667/) (2022)
- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [Generative Modeling by Estimating Gradients of the Data Distribution](https://mlsanity.com/paper/1907.05600/) (2019)
- [GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models](https://mlsanity.com/paper/2112.10741/) (2021)
- [Analyzing and Improving the Image Quality of StyleGAN](https://mlsanity.com/paper/1912.04958/) (2019)
- [Taming Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2012.09841/) (2020)
- [Wide Residual Networks](https://mlsanity.com/paper/1605.07146/) (2016)
- [Perceptual Losses for Real-Time Style Transfer and Super-Resolution](https://mlsanity.com/paper/1603.08155/) (2016)
- [Large Scale GAN Training for High Fidelity Natural Image Synthesis](https://mlsanity.com/paper/1809.11096/) (2018)
- [Arbitrary Style Transfer in Real-time with Adaptive Instance Normalization](https://mlsanity.com/paper/1703.06868/) (2017)
- [Glow: Generative Flow with Invertible 1x1 Convolutions](https://mlsanity.com/paper/1807.03039/) (2018)
- [Generating Diverse High-Fidelity Images with VQ-VAE-2](https://mlsanity.com/paper/1906.00446/) (2019)
- [Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks](https://mlsanity.com/paper/2004.06165/) (2020)
- [StyleCLIP: Text-Driven Manipulation of StyleGAN Imagery](https://mlsanity.com/paper/2103.17249/) (2021)
- [Vector Quantized Diffusion Model for Text-to-Image Synthesis](https://mlsanity.com/paper/2111.14822/) (2021)
- [AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networks](https://mlsanity.com/paper/1711.10485/) (2017)
- [DiffusionCLIP: Text-Guided Diffusion Models for Robust Image Manipulation](https://mlsanity.com/paper/2110.02711/) (2021)
- [ILVR: Conditioning Method for Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2108.02938/) (2021)

## Built on by (most cited first, 30 of 30)

- [Inversion-Based Style Transfer with Diffusion Models](https://mlsanity.com/paper/2211.13203/) (2022)
- [Text-to-image Diffusion Models in Generative AI: A Survey](https://mlsanity.com/paper/2303.07909/) (2023)
- [ProSpect: Prompt Spectrum for Attribute-Aware Personalization of Diffusion Models](https://mlsanity.com/paper/2305.16225/) (2023)
- [Text2Earth: Unlocking Text-driven Remote Sensing Image Generation with a Global-Scale Dataset and a Foundation Model](https://mlsanity.com/paper/2501.00895/) (2025)
- [DiffStyler: Controllable Dual Diffusion for Text-Driven Image Stylization](https://mlsanity.com/paper/2211.10682/) (2022)
- [DreamTuner: Single Image is Enough for Subject-Driven Generation](https://mlsanity.com/paper/2312.13691/) (2023)
- [High-Fidelity Guided Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2211.17084/) (2022)
- [ArtBank: Artistic Style Transfer with Pre-trained Diffusion Model and Implicit Style Prompt Bank](https://mlsanity.com/paper/2312.06135/) (2023)
- [Region-Aware Diffusion for Zero-shot Text-driven Image Editing](https://mlsanity.com/paper/2302.11797/) (2023)
- [FreeStyle: Free Lunch for Text-guided Style Transfer using Diffusion Models](https://mlsanity.com/paper/2401.15636/) (2024)
- [3DStyle-Diffusion: Pursuing Fine-grained Text-driven 3D Stylization with 2D Diffusion Models](https://mlsanity.com/paper/2311.05464/) (2023)
- [Diffusion-Based Visual Art Creation: A Survey and New Perspectives](https://mlsanity.com/paper/2408.12128/) (2024)
- [GRAPHGPT-O: Synergistic Multimodal Comprehension and Generation on Graphs](https://mlsanity.com/paper/2502.11925/) (2025)
- [Towards Highly Realistic Artistic Style Transfer via Stable Diffusion with Step-aware and Layer-aware Prompt](https://mlsanity.com/paper/2404.11474/) (2024)
- [Style-A-Video: Agile Diffusion for Arbitrary Text-based Video Style Transfer](https://mlsanity.com/paper/2305.05464/) (2023)
- [MLaGA: Multimodal Large Language and Graph Assistant](https://mlsanity.com/paper/2506.02568/) (2025)
- [A Unified Arbitrary Style Transfer Framework via Adaptive Contrastive Learning](https://mlsanity.com/paper/2303.12710/) (2023)
- [Diverse Diffusion: Enhancing Image Diversity in Text-to-Image Generation](https://mlsanity.com/paper/2310.12583/) (2023)
- [Creativity in AI: Progresses and Challenges](https://mlsanity.com/paper/2410.17218/) (2024)
- [CreativeSynth: Cross-Art-Attention for Artistic Image Synthesis with Multimodal Diffusion](https://mlsanity.com/paper/2401.14066/) (2024)
- [PARASOL: Parametric Style Control for Diffusion Image Synthesis](https://mlsanity.com/paper/2303.06464/) (2023)
- [StyleMamba : State Space Model for Efficient Text-driven Image Style Transfer](https://mlsanity.com/paper/2405.05027/) (2024)
- [Generative AI in Depth: A Survey of Recent Advances, Model Variants, and Real-World Applications](https://mlsanity.com/paper/2510.21887/) (2025)
- [Multimodal Classification via Modal-Aware Interactive Enhancement](https://mlsanity.com/paper/2407.04587/) (2024)
- [Beyond Inserting: Learning Identity Embedding for Semantic-Fidelity Personalized Diffusion Generation](https://mlsanity.com/paper/2402.00631/) (2024)
- [ExactDreamer: High-Fidelity Text-to-3D Content Creation via Exact Score Matching](https://mlsanity.com/paper/2405.15914/) (2024)
- [PT-Mark: Invisible Watermarking for Text-to-image Diffusion Models via Semantic-aware Pivotal Tuning](https://mlsanity.com/paper/2504.10853/) (2025)
- [KLDD: Kalman Filter based Linear Deformable Diffusion Model in Retinal Image Segmentation](https://mlsanity.com/paper/2410.02808/) (2024)
- [LoyalDiffusion: A Diffusion Model Guarding Against Data Replication](https://mlsanity.com/paper/2412.01118/) (2024)
- [Bridging Text and Image for Artist Style Transfer via Contrastive Learning](https://mlsanity.com/paper/2410.09566/) (2024)

## Abstract (arXiv)

Digital art synthesis is receiving increasing attention in the multimedia community because of engaging the public with art effectively. Current digital art synthesis methods usually use single-modality inputs as guidance, thereby limiting the expressiveness of the model and the diversity of generated results. To solve this problem, we propose the multimodal guided artwork diffusion (MGAD) model, which is a diffusion-based digital artwork generation approach that utilizes multimodal prompts as guidance to control the classifier-free diffusion model. Additionally, the contrastive language-image pretraining (CLIP) model is used to unify text and image modalities. Extensive experimental results on the quality and quantity of the generated digital art paintings confirm the effectiveness of the combination of the diffusion model and multimodal guidance. Code is available at https://github.com/haha-lisa/MGAD-multimodal-guided-artwork-diffusion.