mlsanityData to 14 Oct 2026

Visual Style Prompting with Swapping Self-Attention

arXiv 2402.12974 ↗First posted 20 Feb 2024cs.CVcited by 63

On the map: Diffusion models → Image generation, text-to-image and editing

Abstract (arXiv)

In the evolving domain of text-to-image generation, diffusion models have emerged as powerful tools in content creation. Despite their remarkable capability, existing models still face challenges in achieving controlled generation with a consistent style, requiring costly fine-tuning or often inadequately transferring the visual elements due to content leakage. To address these challenges, we propose a novel approach, ours, to produce a diverse range of images while maintaining specific style elements and nuances. During the denoising process, we keep the query from original features while swapping the key and value with those from reference features in the late self-attention layers. This approach allows for the visual style prompting without any fine-tuning, ensuring that generated images maintain a faithful style. Through extensive evaluation across various styles and text prompts, our method demonstrates superiority over existing approaches, best reflecting the style of the references and ensuring that resulting images match the text prompts most accurately. Our project page is available https://curryjung.github.io/VisualStylePrompt/.

Key facts. Cited by 63 papers in the map (data through 14 Oct 2026). Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models. Builds on 33 of its 41 references that are in the map. First posted on arXiv 20 Feb 2024.

Builds on

33 of 41 references.

Closest ancestors · 7

Same direction, newest first

  1. 2023IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Models13 Aug
  2. 2023Highly Personalized Text Embedding for Image Manipulation by Stable Diffusion15 Mar
  3. 2023Adding Conditional Control to Text-to-Image Diffusion Models10 Feb
  4. 2022Multi-Concept Customization of Text-to-Image Diffusion8 Dec
  5. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation25 Aug
  6. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding23 May
  7. 2022Hierarchical Text-Conditional Image Generation with CLIP Latents13 Apr

Other references · 26

  1. 2017Attention Is All You Needcited by 60,361
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  5. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,177
  6. 2020Denoising Diffusion Implicit Modelscited by 9,781
  7. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 9,122
  8. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  9. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,356
  10. 2021Zero-Shot Text-to-Image Generationcited by 4,288
  11. 2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,245
  12. 2017Arbitrary Style Transfer in Real-time with Adaptive Instance Normalizationcited by 2,010
14 more
  1. 2022Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generationcited by 961
  2. 2022Plug-and-Play Diffusion Features for Text-Driven Image-to-Image Translationcited by 906
  3. 2023MasaCtrl: Tuning-Free Mutual Self-Attention Control for Consistent Image Synthesis and Editingcited by 727
  4. 2022Diffusion Models already have a Semantic Latent Spacecited by 328
  5. 2023Rerender A Video: Zero-Shot Text-Guided Video-to-Video Translationcited by 253
  6. 2023P+: Extended Textual Conditioning in Text-to-Image Generationcited by 242
  7. 2023Break-A-Scene: Extracting Multiple Concepts from a Single Imagecited by 231
  8. 2023Style Aligned Image Generation via Shared Attentioncited by 216
  9. 2018Arbitrary Style Transfer with Style-Attentional Networkscited by 198
  10. 2018Avatar-Net: Multi-scale Zero-shot Style Transfer by Feature Decorationcited by 115
  11. 2018NSML: Meet the MLaaS platform with a real-world case studycited by 70
  12. 2023ImageBrush: Learning Visual In-Context Instructions for Exemplar-Based Image Manipulationcited by 59
  13. 2017NSML: A Machine Learning Platform That Enables You to Focus on Your Modelscited by 34
  14. 2023Visual Instruction Inversion: Image Editing via Visual Promptingcited by 26

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 25; 2025: 32; 2026: 6.

63 papers.

Sort:

By area:

  1. 2024InstantStyle: Free Lunch towards Style-Preserving in Text-to-Image Generationcited by 170
  2. 2024Stable Flow: Vital Layers for Training-Free Image Editingcited by 113
  3. 2024InstantStyle-Plus: Style Transfer with Content-Preserving in Text-to-Image Generationcited by 53
  4. 2024RB-Modulation: Training-Free Personalization of Diffusion Models using Stochastic Optimal Controlcited by 48
  5. 2025Attention Distillation: A Unified Approach to Visual Characteristics Transfercited by 33
  6. 2025USO: Unified Style and Subject-Driven Generation via Disentangled and Reward Learningcited by 27
  7. 2024StyleStudio: Text-Driven Style Transfer with Selective Control of Style Elementscited by 26
  8. 2024From Parts to Whole: A Unified Reference Framework for Controllable Human Image Generationcited by 17
  9. 2025Exploring Multimodal Diffusion Transformers for Enhanced Prompt-based Image Editingcited by 17
  10. 2025Dance Like a Chicken: Low-Rank Stylization for Human Motion Diffusioncited by 16
  11. 2024MagicTailor: Component-Controllable Personalization in Text-to-Image Diffusion Modelscited by 14
  12. 2024StyleTex: Style Image-Guided Texture Generation for 3D Modelscited by 14
38 more
  1. 2025Enhancing Creative Generation on Stable Diffusion-based Modelscited by 13
  2. 2024FAM Diffusion: Frequency and Attention Modulation for High-Resolution Image Generation with Stable Diffusioncited by 12
  3. 2024Conditional Balance: Improving Multi-Conditioning Trade-Offs in Image Generationcited by 9
  4. 2025AttenST: A Training-Free Attention-Driven Style Transfer Framework with Pre-Trained Diffusion Modelscited by 9
  5. 2025ConsisLoRA: Enhancing Content and Style Consistency for LoRA-based Style Transfercited by 9
  6. 2025QR-LoRA: Efficient and Disentangled Fine-tuning via QR Decomposition for Customized Generationcited by 9
  7. 2024TV-3DG: Mastering Text-to-3D Customized Generation with Visual Promptcited by 8
  8. 2024Style3D: Attention-guided Multi-view Style Transfer for 3D Object Generationcited by 8
  9. 2025Free-Lunch Color-Texture Disentanglement for Stylized Image Generationcited by 8
  10. 20253D Stylization via Large Reconstruction Modelcited by 8
  11. 2026The Spatial Blindspot of Vision-Language Modelscited by 8
  12. 2024An Efficient and Harmonized Framework for Balanced Cross-Domain Feature Integrationcited by 7
  13. 2024AttenCraft: Attention-guided Disentanglement of Multiple Concepts for Text-to-Image Customizationcited by 6
  14. 2024pOps: Photo-Inspired Diffusion Operatorscited by 6
  15. 2024ReEdit: Multimodal Exemplar-Based Image Editing with Diffusion Modelscited by 6
  16. 2025Style Transfer with Diffusion Models for Synthetic-to-Real Domain Adaptationcited by 6
  17. 2024PixelMan: Consistent Object Editing with Diffusion Models via Pixel Manipulation and Generationcited by 5
  18. 2025A Comprehensive Survey on Visual Concept Mining in Text-to-image Diffusion Modelscited by 5
  19. 2024Dream-in-Style: Text-to-3D Generation Using Stylized Score Distillationcited by 4
  20. 2025ArtCrafter: Text-Image Aligning Style Transfer via Embedding Reframingcited by 4
  21. 2025StyleAR: Customizing Multimodal Autoregressive Model for Style-Aligned Text-to-Image Generationcited by 4
  22. 2024OmniPrism: Learning Disentangled Visual Concept for Image Generationcited by 3
  23. 2025MixSA: Training-free Reference-based Sketch Extraction via Mixture-of-Self-Attentioncited by 3
  24. 2025IP-Prompter: Training-Free Theme-Specific Image Generation via Dynamic Visual Promptingcited by 3
  25. 2024FunEditor: Achieving Complex Image Edits via Function Aggregation with Diffusion Modelscited by 2
  26. 2025SPG: Improving Motion Diffusion by Smooth Perturbation Guidancecited by 2
  27. 2025Inversion-Free Video Style Transfer with Trajectory Reset Attention Control and Content-Style Bridgingcited by 2
  28. 2025TransLight: Image-Guided Customized Lighting Control with Generative Decouplingcited by 2
  29. 2025Category-Aware 3D Object Composition with Disentangled Texture and Shape Multi-view Diffusioncited by 2
  30. 2026Scale-Separated Conditioning for Style-Encoder-Free Diffusion Stylizationcited by 2
  31. 2024DECOR:Decomposition and Projection of Text Embeddings for Text-to-Image Customizationcited by 1
  32. 2024CA-Edit: Causality-Aware Condition Adapter for High-Fidelity Local Facial Attribute Editingcited by 1
  33. 2024WikiStyle+: A Multimodal Approach to Content-Style Representation Disentanglement for Artistic Image Stylizationcited by 1
  34. 2025Semantix: An Energy Guided Sampler for Semantic Style Transfercited by 1
  35. 2025Only-Style: Stylistic Consistency in Image Generation without Content Leakagecited by 1
  36. 2025Jigsaw3D: Disentangled 3D Style Transfer via Patch Shuffling and Maskingcited by 1
  37. 2025Stroke2Sketch: Harnessing Stroke Attributes for Training-Free Sketch Generationcited by 1
  38. 2026FreeStyle: Free Control of Style-Content Dual-Reference Generation from Community LoRA Miningcited by 1

50 most cited of 63 All 63 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023ControlStyle: Text-Driven Stylized Image Generation Using Diffusion Priors
  2. 2025StyleKeeper: Prevent Content Leakage using Negative Visual Query Guidance
  3. 2024ArtWeaver: Advanced Dynamic Style Integration via Diffusion Model
  4. 2026PokeFusion Attention: A Lightweight Cross-Attention Mechanism for Style-Conditioned Image Generation
  5. 2025StyleBlend: Enhancing Style-Specific Content Creation in Text-to-Image Diffusion Models
  6. 2022Text-Guided Synthesis of Artistic Images with Retrieval-Augmented Diffusion Models
  7. 2024StyleTokenizer: Defining Image Style by a Single Instance for Controlling Diffusion Models
  8. 2023Style Injection in Diffusion: A Training-free Approach for Adapting Large-scale Diffusion Models for Style Transfer