mlsanityData to 7 Oct 2026

Draw Your Art Dream: Diverse Digital Art Synthesis with Multimodal Guided Diffusion

arXiv 2209.13360 ↗First posted 27 Sep 2022cs.CVcited by 30

On the map: Diffusion models → Image generation, text-to-image and editing

Abstract (arXiv)

Digital art synthesis is receiving increasing attention in the multimedia community because of engaging the public with art effectively. Current digital art synthesis methods usually use single-modality inputs as guidance, thereby limiting the expressiveness of the model and the diversity of generated results. To solve this problem, we propose the multimodal guided artwork diffusion (MGAD) model, which is a diffusion-based digital artwork generation approach that utilizes multimodal prompts as guidance to control the classifier-free diffusion model. Additionally, the contrastive language-image pretraining (CLIP) model is used to unify text and image modalities. Extensive experimental results on the quality and quantity of the generated digital art paintings confirm the effectiveness of the combination of the diffusion model and multimodal guidance. Code is available at https://github.com/haha-lisa/MGAD-multimodal-guided-artwork-diffusion.

Key facts. Cited by 30 papers in the map (data through 7 Oct 2026). Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models. Builds on 53 of its 76 references that are in the map. First posted on arXiv 27 Sep 2022.

Builds on

53 of 76 references.

Closest ancestors · 4

Same direction, newest first

  1. 2022Hierarchical Text-Conditional Image Generation with CLIP Latents13 Apr
  2. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models20 Dec
  3. 2021More Control for Free! Image Synthesis with Semantic Diffusion Guidance10 Dec
  4. 2021DiffusionCLIP: Text-Guided Diffusion Models for Robust Image Manipulation6 Oct

Other references · 49

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  5. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,029
  6. 2020Denoising Diffusion Implicit Modelscited by 9,655
  7. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  8. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  9. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  10. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 5,986
  11. 2022Classifier-Free Diffusion Guidancecited by 5,717
  12. 2022Generative Adversarial Networkscited by 5,603
37 more
  1. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  2. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,067
  3. 2019Analyzing and Improving the Image Quality of StyleGANcited by 3,580
  4. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
  5. 2016Wide Residual Networkscited by 3,057
  6. 2016Perceptual Losses for Real-Time Style Transfer and Super-Resolutioncited by 2,678
  7. 2018Large Scale GAN Training for High Fidelity Natural Image Synthesiscited by 2,609
  8. 2017Arbitrary Style Transfer in Real-time with Adaptive Instance Normalizationcited by 1,998
  9. 2018Glow: Generative Flow with Invertible 1x1 Convolutionscited by 1,539
  10. 2019Generating Diverse High-Fidelity Images with VQ-VAE-2cited by 1,376
  11. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
  12. 2021StyleCLIP: Text-Driven Manipulation of StyleGAN Imagerycited by 990
  13. 2021Vector Quantized Diffusion Model for Text-to-Image Synthesiscited by 773
  14. 2017AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networkscited by 664
  15. 2021ILVR: Conditioning Method for Denoising Diffusion Probabilistic Modelscited by 634
  16. 2020VirTex: Learning Visual Representations from Textual Annotationscited by 338
  17. 2021CLIPstyler: Image Style Transfer with a Single Text Conditioncited by 203
  18. 2021StyTr2: Image Style Transfer with Transformerscited by 186
  19. 2021CLIPDraw: Exploring Text-to-Drawing Synthesis through Language-Image Encoderscited by 186
  20. 2021AdaAttN: Revisit Attention Mechanism in Arbitrary Neural Style Transfercited by 178
  21. 2021StyleGAN-NADA: CLIP-Guided Domain Adaptation of Image Generatorscited by 164
  22. 2020Adversarial score matching and improved sampling for image generationcited by 95
  23. 2022Domain Enhanced Arbitrary Image Style Transfer via Contrastive Learningcited by 93
  24. 2020Arbitrary Video Style Transfer via Multi-Channel Correlationcited by 84
  25. 2018Generating High Fidelity Images with Subscale Pixel Networks and Multidimensional Upscalingcited by 74
  26. 2021FuseDream: Training-Free Text-to-Image Generation with Improved CLIP+GAN Space Optimizationcited by 71
  27. 2020Arbitrary Style Transfer via Multi-Adaptation Networkcited by 70
  28. 2021DAE-GAN: Dynamic Aspect-aware GAN for Text-to-Image Synthesiscited by 45
  29. 2019Diversified Arbitrary Style Transfer via Deep Feature Perturbationcited by 42
  30. 2020Describe What to Change: A Text-guided Unsupervised Image-to-Image Translation Approachcited by 29
  31. 2022StyleCLIPDraw: Coupling Content and Style in Text-to-Drawing Translationcited by 28
  32. 2017Multi-View Image Generation from a Single-Viewcited by 27
  33. 2021Cycle-Consistent Inverse GAN for Text-to-Image Synthesiscited by 27
  34. 2017ArtGAN: Artwork Synthesis with Conditional Categorical GANscited by 17
  35. 2020Distribution Aligned Multimodal and Multi-Domain Image Stylizationcited by 10
  36. 2021dualFace:Two-Stage Drawing Guidance for Freehand Portrait Sketchingcited by 8
  37. 2021Deep Image Synthesis from Intuitive User Input: A Review and Perspectivescited by 8

Built on by

Cited by year

Citing papers per year: 2022: 3; 2023: 10; 2024: 12; 2025: 5.

30 papers.

Sort:

By area:

  1. 2022Inversion-Based Style Transfer with Diffusion Modelscited by 290
  2. 2023Text-to-image Diffusion Models in Generative AI: A Surveycited by 264
  3. 2023ProSpect: Prompt Spectrum for Attribute-Aware Personalization of Diffusion Modelscited by 75
  4. 2025Text2Earth: Unlocking Text-driven Remote Sensing Image Generation with a Global-Scale Dataset and a Foundation Modelcited by 46
  5. 2022DiffStyler: Controllable Dual Diffusion for Text-Driven Image Stylizationcited by 45
  6. 2023DreamTuner: Single Image is Enough for Subject-Driven Generationcited by 41
  7. 2022High-Fidelity Guided Image Synthesis with Latent Diffusion Modelscited by 36
  8. 2023ArtBank: Artistic Style Transfer with Pre-trained Diffusion Model and Implicit Style Prompt Bankcited by 36
  9. 2023Region-Aware Diffusion for Zero-shot Text-driven Image Editingcited by 25
  10. 2024FreeStyle: Free Lunch for Text-guided Style Transfer using Diffusion Modelscited by 21
  11. 20233DStyle-Diffusion: Pursuing Fine-grained Text-driven 3D Stylization with 2D Diffusion Modelscited by 17
  12. 2024Diffusion-Based Visual Art Creation: A Survey and New Perspectivescited by 17
18 more
  1. 2025GRAPHGPT-O: Synergistic Multimodal Comprehension and Generation on Graphscited by 17
  2. 2024Towards Highly Realistic Artistic Style Transfer via Stable Diffusion with Step-aware and Layer-aware Promptcited by 16
  3. 2023Style-A-Video: Agile Diffusion for Arbitrary Text-based Video Style Transfercited by 14
  4. 2025MLaGA: Multimodal Large Language and Graph Assistantcited by 12
  5. 2023A Unified Arbitrary Style Transfer Framework via Adaptive Contrastive Learningcited by 10
  6. 2023Diverse Diffusion: Enhancing Image Diversity in Text-to-Image Generationcited by 8
  7. 2024Creativity in AI: Progresses and Challengescited by 7
  8. 2024CreativeSynth: Cross-Art-Attention for Artistic Image Synthesis with Multimodal Diffusioncited by 6
  9. 2023PARASOL: Parametric Style Control for Diffusion Image Synthesiscited by 5
  10. 2024StyleMamba : State Space Model for Efficient Text-driven Image Style Transfercited by 5
  11. 2025Generative AI in Depth: A Survey of Recent Advances, Model Variants, and Real-World Applicationscited by 5
  12. 2024Multimodal Classification via Modal-Aware Interactive Enhancementcited by 4
  13. 2024Beyond Inserting: Learning Identity Embedding for Semantic-Fidelity Personalized Diffusion Generationcited by 2
  14. 2024ExactDreamer: High-Fidelity Text-to-3D Content Creation via Exact Score Matchingcited by 2
  15. 2025PT-Mark: Invisible Watermarking for Text-to-image Diffusion Models via Semantic-aware Pivotal Tuningcited by 2
  16. 2024KLDD: Kalman Filter based Linear Deformable Diffusion Model in Retinal Image Segmentationcited by 1
  17. 2024LoyalDiffusion: A Diffusion Model Guarding Against Data Replicationcited by 1
  18. 2024Bridging Text and Image for Artist Style Transfer via Contrastive Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Draw Like an Artist: Complex Scene Generation with Diffusion Model via Composition, Painting, and Retouching
  2. 2022Text-Guided Synthesis of Artistic Images with Retrieval-Augmented Diffusion Models
  3. 2023Stable Diffusion Reference Only: Image Prompt and Blueprint Jointly Guided Multi-Condition Diffusion Model for Secondary Painting
  4. 2022Arbitrary Style Guidance for Enhanced Diffusion-Based Text-to-Image Generation
  5. 2026Creative Image Generation with Diffusion Models
  6. 2023Diffusion Brush: A Latent Diffusion Model-based Editing Tool for AI-generated Images
  7. 2023Composite Diffusion | whole >= Sigma parts
  8. 2022The Stable Artist: Steering Semantics in Diffusion Latent Space