# ArtBank: Artistic Style Transfer with Pre-trained Diffusion Model and Implicit Style Prompt Bank

arXiv: https://arxiv.org/abs/2312.06135 · HTML: https://mlsanity.com/paper/2312.06135/

## Key facts

- Cited by 36 papers in the map (data through 14 Oct 2026).
- Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models.
- Builds on 28 of its 44 references that are in the map.
- First posted on arXiv 11 Dec 2023.

## Placed in

- Diffusion models / [Image generation, text-to-image and editing](https://mlsanity.com/diffusion/image-generation-text-to-image-and-editing/)

## Builds on (most cited first, 28 of 28)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Improved Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2102.09672/) (2021)
- [GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models](https://mlsanity.com/paper/2112.10741/) (2021)
- [Prompt-to-Prompt Image Editing with Cross Attention Control](https://mlsanity.com/paper/2208.01626/) (2022)
- [CLIPScore: A Reference-free Evaluation Metric for Image Captioning](https://mlsanity.com/paper/2104.08718/) (2021)
- [Arbitrary Style Transfer in Real-time with Adaptive Instance Normalization](https://mlsanity.com/paper/1703.06868/) (2017)
- [Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks](https://mlsanity.com/paper/1703.10593/) (2017)
- [Universal Style Transfer via Feature Transforms](https://mlsanity.com/paper/1705.08086/) (2017)
- [BoxDiff: Text-to-Image Synthesis with Training-Free Box-Constrained Diffusion](https://mlsanity.com/paper/2307.10816/) (2023)
- [Learning to Discover Cross-Domain Relations with Generative Adversarial Networks](https://mlsanity.com/paper/1703.05192/) (2017)
- [Splicing ViT Features for Semantic Appearance Transfer](https://mlsanity.com/paper/2201.00424/) (2022)
- [Arbitrary Style Transfer with Style-Attentional Networks](https://mlsanity.com/paper/1812.02342/) (2018)
- [U-GAT-IT: Unsupervised Generative Attentional Networks with Adaptive Layer-Instance Normalization for Image-to-Image Translation](https://mlsanity.com/paper/1907.10830/) (2019)
- [AdaAttN: Revisit Attention Mechanism in Arbitrary Neural Style Transfer](https://mlsanity.com/paper/2108.03647/) (2021)
- [Diffusion-based Image Translation using Disentangled Style and Content Representation](https://mlsanity.com/paper/2209.15264/) (2022)
- [Geometry-Consistent Generative Adversarial Networks for One-Sided Unsupervised Domain Mapping](https://mlsanity.com/paper/1809.05852/) (2018)
- [Expressive Text-to-Image Generation with Rich Text](https://mlsanity.com/paper/2304.06720/) (2023)
- [The Spatially-Correlative Loss for Various Image Translation Tasks](https://mlsanity.com/paper/2104.00854/) (2021)
- [A Style-Aware Content Loss for Real-time HD Style Transfer](https://mlsanity.com/paper/1807.10201/) (2018)
- [Draw Your Art Dream: Diverse Digital Art Synthesis with Multimodal Guided Diffusion](https://mlsanity.com/paper/2209.13360/) (2022)
- [AesUST: Towards Aesthetic-Enhanced Universal Style Transfer](https://mlsanity.com/paper/2208.13016/) (2022)
- [VGOS: Voxel Grid Optimization for View Synthesis from Sparse Inputs](https://mlsanity.com/paper/2304.13386/) (2023)
- [ITTR: Unpaired Image-to-Image Translation with Transformers](https://mlsanity.com/paper/2203.16015/) (2022)
- [Creative Painting with Latent Diffusion Models](https://mlsanity.com/paper/2209.14697/) (2022)

## Built on by (most cited first, 30 of 36)

- [AddSR: Accelerating Diffusion-based Blind Super-Resolution with Adversarial Diffusion Distillation](https://mlsanity.com/paper/2404.01717/) (2024)
- [OmniStyle: Filtering High Quality Style Transfer Data at Scale](https://mlsanity.com/paper/2505.14028/) (2025)
- [RelaCtrl: Relevance-Guided Efficient Control for Diffusion Transformers](https://mlsanity.com/paper/2502.14377/) (2025)
- [PlanGen: Towards Unified Layout Planning and Image Generation in Auto-Regressive Vision Language Models](https://mlsanity.com/paper/2503.10127/) (2025)
- [Diffusion-Based Visual Art Creation: A Survey and New Perspectives](https://mlsanity.com/paper/2408.12128/) (2024)
- [Towards Highly Realistic Artistic Style Transfer via Stable Diffusion with Step-aware and Layer-aware Prompt](https://mlsanity.com/paper/2404.11474/) (2024)
- [UniVST: A Unified Framework for Training-free Localized Video Style Transfer](https://mlsanity.com/paper/2410.20084/) (2024)
- [FancyVideo: Towards Dynamic and Consistent Video Generation via Cross-frame Textual Guidance](https://mlsanity.com/paper/2408.08189/) (2024)
- [Uni-Layout: Integrating Human Feedback in Unified Layout Generation and Evaluation](https://mlsanity.com/paper/2508.02374/) (2025)
- [U-StyDiT: Ultra-high Quality Artistic Style Transfer Using Diffusion Transformers](https://mlsanity.com/paper/2503.08157/) (2025)
- [Rethinking Video Deblurring with Wavelet-Aware Dynamic Transformer and Diffusion Model](https://mlsanity.com/paper/2408.13459/) (2024)
- [ConsisLoRA: Enhancing Content and Style Consistency for LoRA-based Style Transfer](https://mlsanity.com/paper/2503.10614/) (2025)
- [Lay2Story: Extending Diffusion Transformers for Layout-Togglable Story Generation](https://mlsanity.com/paper/2508.08949/) (2025)
- [MoFu: Scale-Aware Modulation and Fourier Fusion for Multi-Subject Video Generation](https://mlsanity.com/paper/2512.22310/) (2025)
- [InnoAds-Composer: Efficient Condition Composition for E-Commerce Poster Generation](https://mlsanity.com/paper/2603.05898/) (2026)
- [HiFi-Inpaint: Towards High-Fidelity Reference-Based Inpainting for Generating Detail-Preserving Human-Product Images](https://mlsanity.com/paper/2603.02210/) (2026)
- [ArtCrafter: Text-Image Aligning Style Transfer via Embedding Reframing](https://mlsanity.com/paper/2501.02064/) (2025)
- [SCSA: A Plug-and-Play Semantic Continuous-Sparse Attention for Arbitrary Semantic Style Transfer](https://mlsanity.com/paper/2503.04119/) (2025)
- [Rethink Arbitrary Style Transfer with Transformer and Contrastive Learning](https://mlsanity.com/paper/2404.13584/) (2024)
- [DyArtbank: Diverse Artistic Style Transfer via Pre-trained Stable Diffusion and Dynamic Style Prompt Artbank](https://mlsanity.com/paper/2503.08392/) (2025)
- [SPAST: Arbitrary Style Transfer with Style Priors via Pre-trained Large-scale Model](https://mlsanity.com/paper/2505.08695/) (2025)
- [Restore Text First, Enhance Image Later: Two-Stage Scene Text Image Super-Resolution with Glyph Structure Guidance](https://mlsanity.com/paper/2510.21590/) (2025)
- [StyMam: A Mamba-Based Generator for Artistic Style Transfer](https://mlsanity.com/paper/2601.12954/) (2026)
- [Large Generative Graph Models](https://mlsanity.com/paper/2406.05109/) (2024)
- [Why Compress What You Can Generate? When GPT-4o Generation Ushers in Image Compression Fields](https://mlsanity.com/paper/2504.21814/) (2025)
- [RoSe: Robust Self-supervised Stereo Matching under Adverse Weather Conditions](https://mlsanity.com/paper/2509.19165/) (2025)
- [PixelShuffler: A Simple Image Translation Through Pixel Rearrangement](https://mlsanity.com/paper/2410.03021/) (2024)
- [WikiStyle+: A Multimodal Approach to Content-Style Representation Disentanglement for Artistic Image Stylization](https://mlsanity.com/paper/2412.14496/) (2024)
- [The Art That Poses Back: Assessing AI Pastiches after Contemporary Artworks](https://mlsanity.com/paper/2603.06324/) (2026)
- [PIPBench: A Profile-Inclusive Framework for Personalized Image Generation Evaluation](https://mlsanity.com/paper/2607.06440/) (2026)

## Abstract (arXiv)

Artistic style transfer aims to repaint the content image with the learned artistic style. Existing artistic style transfer methods can be divided into two categories: small model-based approaches and pre-trained large-scale model-based approaches. Small model-based approaches can preserve the content strucuture, but fail to produce highly realistic stylized images and introduce artifacts and disharmonious patterns; Pre-trained large-scale model-based approaches can generate highly realistic stylized images but struggle with preserving the content structure. To address the above issues, we propose ArtBank, a novel artistic style transfer framework, to generate highly realistic stylized images while preserving the content structure of the content images. Specifically, to sufficiently dig out the knowledge embedded in pre-trained large-scale models, an Implicit Style Prompt Bank (ISPB), a set of trainable parameter matrices, is designed to learn and store knowledge from the collection of artworks and behave as a visual prompt to guide pre-trained large-scale models to generate highly realistic stylized images while preserving content structure. Besides, to accelerate training the above ISPB, we propose a novel Spatial-Statistical-based self-Attention Module (SSAM). The qualitative and quantitative experiments demonstrate the superiority of our proposed method over state-of-the-art artistic style transfer methods.