mlsanityData to 7 Oct 2026

ControlStyle: Text-Driven Stylized Image Generation Using Diffusion Priors

arXiv 2311.05463 ↗First posted 9 Nov 2023cs.CV cs.MMcited by 40

On the map: Diffusion models → Image generation, text-to-image and editing

Abstract (arXiv)

Recently, the multimedia community has witnessed the rise of diffusion models trained on large-scale multi-modal data for visual content creation, particularly in the field of text-to-image generation. In this paper, we propose a new task for ``stylizing'' text-to-image models, namely text-driven stylized image generation, that further enhances editability in content creation. Given input text prompt and style image, this task aims to produce stylized images which are both semantically relevant to input text prompt and meanwhile aligned with the style image in style. To achieve this, we present a new diffusion model (ControlStyle) via upgrading a pre-trained text-to-image model with a trainable modulation network enabling more conditions of text prompts and style images. Moreover, diffusion style and content regularizations are simultaneously introduced to facilitate the learning of this modulation network with these diffusion priors, pursuing high-quality stylized text-to-image generation. Extensive experiments demonstrate the effectiveness of our ControlStyle in producing more visually pleasing and artistic results, surpassing a simple combination of text-to-image model and conventional style transfer techniques.

Key facts. Cited by 40 papers in the map (data through 7 Oct 2026). Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models. Builds on 34 of its 44 references that are in the map. First posted on arXiv 9 Nov 2023.

Builds on

34 of 44 references.

Closest ancestors · 6

Same direction, newest first

  1. 2023Adding Conditional Control to Text-to-Image Diffusion Models10 Feb
  2. 2022Inversion-Based Style Transfer with Diffusion Models23 Nov
  3. 2022InstructPix2Pix: Learning to Follow Image Editing Instructions17 Nov
  4. 2022eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoisers2 Nov
  5. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding23 May
  6. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models20 Dec

Other references · 28

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  6. 2020Denoising Diffusion Implicit Modelscited by 9,655
  7. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  8. 2022Generative Adversarial Networkscited by 5,603
  9. 2016Image-to-Image Translation with Conditional Adversarial Networkscited by 4,510
  10. 2022DreamFusion: Text-to-3D using 2D Diffusioncited by 3,006
  11. 2016Perceptual Losses for Real-Time Style Transfer and Super-Resolutioncited by 2,678
  12. 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,409
16 more
  1. 2014Conditional Generative Adversarial Netscited by 2,152
  2. 2017Arbitrary Style Transfer in Real-time with Adaptive Instance Normalizationcited by 1,998
  3. 2022Imagen Video: High Definition Video Generation with Diffusion Modelscited by 1,765
  4. 2017High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANscited by 1,176
  5. 2020Contrastive Learning for Unpaired Image-to-Image Translationcited by 637
  6. 2023Make-It-3D: High-Fidelity 3D Creation from A Single Image with Diffusion Priorcited by 341
  7. 2021CLIPstyler: Image Style Transfer with a Single Text Conditioncited by 203
  8. 2021StyTr2: Image Style Transfer with Transformerscited by 186
  9. 2022Domain Enhanced Arbitrary Image Style Transfer via Contrastive Learningcited by 93
  10. 2021Contextual Transformer Networks for Visual Recognitioncited by 90
  11. 2018To Create What You Tell: Generating Videos from Captionscited by 84
  12. 2022Wave-ViT: Unifying Wavelet and Transformers for Visual Representation Learningcited by 78
  13. 2013Recognizing Image Stylecited by 64
  14. 2019Mocycle-GAN: Unpaired Video-to-Video Translationcited by 37
  15. 2023CAP-VSTNet: Content Affinity Preserved Versatile Style Transfercited by 24
  16. 2023ITstyler: Image-optimized Text-based Style Transfercited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 20; 2025: 15; 2026: 3.

40 papers.

Sort:

By area:

  1. 2024DreamBench++: A Human-Aligned Benchmark for Personalized Image Generationcited by 131
  2. 2024StyleShot: A Snapshot on Any Stylecited by 60
  3. 2024Boosting Diffusion Models with Moving Average Sampling in Frequency Domaincited by 53
  4. 2024SD-DiT: Unleashing the Power of Self-supervised Discrimination in Diffusion Transformercited by 47
  5. 2025Attention Distillation: A Unified Approach to Visual Characteristics Transfercited by 32
  6. 2024LLM4GEN: Leveraging Semantic Representation of LLMs for Text-to-Image Generationcited by 30
  7. 2024VP3D: Unleashing 2D Visual Prompt for Text-to-3D Generationcited by 25
  8. 2024Improving Virtual Try-On with Garment-focused Diffusion Modelscited by 15
  9. 2024SCott: Accelerating Diffusion Models with Stochastic Consistency Distillationcited by 14
  10. 2024StyleTex: Style Image-Guided Texture Generation for 3D Modelscited by 14
  11. 2024Efficient Diffusion as Low Light Enhancercited by 13
  12. 2024Improving Text-guided Object Inpainting with Semantic Pre-inpaintingcited by 12
28 more
  1. 2025ConsisLoRA: Enhancing Content and Style Consistency for LoRA-based Style Transfercited by 9
  2. 2024Block and Detail: Scaffolding Sketch-to-Image Generationcited by 8
  3. 2024Diff-Mosaic: Augmenting Realistic Representations in Infrared Small Target Detection via Diffusion Priorcited by 8
  4. 2025Incorporating Visual Correspondence into Diffusion Model for Virtual Try-Oncited by 8
  5. 2025Hierarchical Masked Autoregressive Models with Low-Resolution Token Pivotscited by 8
  6. 2023X-Dreamer: Creating High-quality 3D Content by Bridging the Domain Gap Between Text-to-2D and Text-to-3D Generationcited by 7
  7. 2024DiffArtist: Towards Structure and Appearance Controllable Image Stylizationcited by 6
  8. 2024FreeEnhance: Tuning-Free Image Enhancement via Content-Consistent Noising-and-Denoising Processcited by 6
  9. 2024Generative Object Insertion in Gaussian Splatting with a Multi-View Diffusion Modelcited by 6
  10. 2026MegaStyle: Constructing Diverse and Scalable Style Dataset via Consistent Text-to-Image Style Mappingcited by 4
  11. 2024Font Style Interpolation with Diffusion Modelscited by 3
  12. 2025Towards Consistent and Controllable Image Synthesis for Face Editingcited by 3
  13. 2025CSHNet: A Novel Information Asymmetric Image Translation Methodcited by 2
  14. 2025StyleBlend: Enhancing Style-Specific Content Creation in Text-to-Image Diffusion Modelscited by 2
  15. 2025Creatively Upscaling Images with Global-Regional Priorscited by 2
  16. 2025Stitch: Training-Free Position Control in Multimodal Diffusion Transformerscited by 2
  17. 2024Extrapolating Prospective Glaucoma Fundus Images through Diffusion Model in Irregular Longitudinal Sequencescited by 1
  18. 2024DomainGallery: Few-shot Domain-driven Image Generation by Attribute-centric Finetuningcited by 1
  19. 2025DiffVQA: Video Quality Assessment Using Diffusion Feature Extractorcited by 1
  20. 2025Exploiting Gaussian Agnostic Representation Learning with Diffusion Priors for Enhanced Infrared Small Target Detectioncited by 1
  21. 2026DreamVAR: Taming Reinforced Visual Autoregressive Model for High-Fidelity Subject-Driven Image Generationcited by 1
  22. 2026Gender Artifacts from Art History to Text-to-Image Generationcited by 1
  23. 2023DXAI: Explaining Classification by Image Decompositioncited by 0
  24. 2024Cross-Domain Image Conversion by CycleDMcited by 0
  25. 2025LLM-Enabled Style and Content Regularization for Personalized Text-to-Image Generationcited by 0
  26. 2025Break Stylistic Sophon: Are We Really Meant to Confine the Imagination in Style Transfer?cited by 0
  27. 2025Do It Yourself (DIY): Modifying Images for Poems in a Zero-Shot Setting Using Weighted Prompt Manipulationcited by 0
  28. 2025Image Generation Based on Image Style Extractioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Training-free Stylized Text-to-Image Generation with Fast Inference
  2. 2024ArtWeaver: Advanced Dynamic Style Integration via Diffusion Model
  3. 2022Arbitrary Style Guidance for Enhanced Diffusion-Based Text-to-Image Generation
  4. 2024Visual Style Prompting with Swapping Self-Attention
  5. 2023Style Injection in Diffusion: A Training-free Approach for Adapting Large-scale Diffusion Models for Style Transfer
  6. 2024StyleTokenizer: Defining Image Style by a Single Instance for Controlling Diffusion Models
  7. 2023Fine-grained Text Style Transfer with Diffusion-Based Language Models
  8. 2024FreeStyle: Free Lunch for Text-guided Style Transfer using Diffusion Models