# Training-Free Multi-Style Fusion Through Reference-Based Adaptive Modulation

arXiv: https://arxiv.org/abs/2509.18602 · HTML: https://mlsanity.com/paper/2509.18602/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models.
- Builds on 31 of its 34 references that are in the map.
- First posted on arXiv 23 Sep 2025.

## Placed in

- Diffusion models / [Image generation, text-to-image and editing](https://mlsanity.com/diffusion/image-generation-text-to-image-and-editing/)

## Builds on (most cited first, 30 of 31)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [Scalable Diffusion Models with Transformers](https://mlsanity.com/paper/2212.09748/) (2022)
- [Squeeze-and-Excitation Networks](https://mlsanity.com/paper/1709.01507/) (2017)
- [CBAM: Convolutional Block Attention Module](https://mlsanity.com/paper/1807.06521/) (2018)
- [Video Diffusion Models](https://mlsanity.com/paper/2204.03458/) (2022)
- [Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2304.08818/) (2023)
- [IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Models](https://mlsanity.com/paper/2308.06721/) (2023)
- [FlowEdit: Inversion-Free Text-Based Editing Using Pre-Trained Flow Models](https://mlsanity.com/paper/2412.08629/) (2024)
- [Style Aligned Image Generation via Shared Attention](https://mlsanity.com/paper/2312.02133/) (2023)
- [CLIPstyler: Image Style Transfer with a Single Text Condition](https://mlsanity.com/paper/2112.00374/) (2021)
- [StyleDrop: Text-to-Image Generation in Any Style](https://mlsanity.com/paper/2306.00983/) (2023)
- [InstantStyle: Free Lunch towards Style-Preserving in Text-to-Image Generation](https://mlsanity.com/paper/2404.02733/) (2024)
- [Semantic Image Inversion and Editing using Rectified Stochastic Differential Equations](https://mlsanity.com/paper/2410.10792/) (2024)
- [Multi-LoRA Composition for Image Generation](https://mlsanity.com/paper/2402.16843/) (2024)
- [Beyond First-Order Tweedie: Solving Inverse Problems using Latent Diffusion](https://mlsanity.com/paper/2312.00852/) (2023)
- [Diffusion Models Without Attention](https://mlsanity.com/paper/2311.18257/) (2023)
- [DEADiff: An Efficient Stylization Diffusion Model with Disentangled Representations](https://mlsanity.com/paper/2403.06951/) (2024)
- [Multi-style Generative Network for Real-time Transfer](https://mlsanity.com/paper/1703.06953/) (2017)
- [RB-Modulation: Training-Free Personalization of Diffusion Models using Stochastic Optimal Control](https://mlsanity.com/paper/2405.17401/) (2024)
- [VT-CLIP: Enhancing Vision-Language Models with Visual-guided Texts](https://mlsanity.com/paper/2112.02399/) (2021)
- [FreeTuner: Any Subject in Any Style with Training-free Diffusion](https://mlsanity.com/paper/2405.14201/) (2024)
- [SEAL: Semantic Attention Learning for Long Video Representation](https://mlsanity.com/paper/2412.01798/) (2024)
- [U-StyDiT: Ultra-high Quality Artistic Style Transfer Using Diffusion Transformers](https://mlsanity.com/paper/2503.08157/) (2025)

## Built on by (most cited first, 7 of 7)

- [ConeSep: Cone-based Robust Noise-Unlearning Compositional Network for Composed Image Retrieval](https://mlsanity.com/paper/2604.20358/) (2026)
- [TEMA: Anchor the Image, Follow the Text for Multi-Modification Composed Image Retrieval](https://mlsanity.com/paper/2604.21806/) (2026)
- [COMBINER: Composed Image Retrieval Guided by Attribute-based Neighbor Relations](https://mlsanity.com/paper/2606.04604/) (2026)
- [Toward Stable Semi-Supervised Remote Sensing Segmentation via Co-Guidance and Co-Fusion](https://mlsanity.com/paper/2512.23035/) (2025)
- [IMAGINE: Adaptive Schema-Imagery Enhanced Composition for Composed Video Retrieval](https://mlsanity.com/paper/2606.08144/) (2026)
- [CARE: Training-Free Controllable Restoration for Medical Images via Dual-Latent Steering](https://mlsanity.com/paper/2603.25026/) (2026)
- [JMT-YOLO: Multi-task Domain Adaptation via Segmentation-Guided Consistency Learning](https://mlsanity.com/paper/2610.12901/) (2026)

## Abstract (arXiv)

We propose Adaptive Multi-Style Fusion (AMSF), a reference-based training-free framework that enables controllable fusion of multiple reference styles in diffusion models. Most of the existing reference-based methods are limited by (a) acceptance of only one style image, thus prohibiting hybrid aesthetics and scalability to more styles, and (b) lack of a principled mechanism to balance several stylistic influences. AMSF mitigates these challenges by encoding all style images and textual hints with a semantic token decomposition module that is adaptively injected into every cross-attention layer of an frozen diffusion model. A similarity-aware re-weighting module then recalibrates, at each denoising step, the attention allocated to every style component, yielding balanced and user-controllable blends without any fine-tuning or external adapters. Both qualitative and quantitative evaluations show that AMSF produces multi-style fusion results that consistently outperform the state-of-the-art approaches, while its fusion design scales seamlessly to two or more styles. These capabilities position AMSF as a practical step toward expressive multi-style generation in diffusion models.