# Balanced Image Stylization with Style Matching Score

arXiv: https://arxiv.org/abs/2503.07601 · HTML: https://mlsanity.com/paper/2503.07601/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models.
- Builds on 31 of its 41 references that are in the map.
- First posted on arXiv 10 Mar 2025.

## Placed in

- Diffusion models / [Image generation, text-to-image and editing](https://mlsanity.com/diffusion/image-generation-text-to-image-and-editing/)

## Builds on (most cited first, 30 of 31)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [Adding Conditional Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.05543/) (2023)
- [Elucidating the Design Space of Diffusion-Based Generative Models](https://mlsanity.com/paper/2206.00364/) (2022)
- [DreamFusion: Text-to-3D using 2D Diffusion](https://mlsanity.com/paper/2209.14988/) (2022)
- [Perceptual Losses for Real-Time Style Transfer and Super-Resolution](https://mlsanity.com/paper/1603.08155/) (2016)
- [Arbitrary Style Transfer in Real-time with Adaptive Instance Normalization](https://mlsanity.com/paper/1703.06868/) (2017)
- [Palette: Image-to-Image Diffusion Models](https://mlsanity.com/paper/2111.05826/) (2021)
- [ProlificDreamer: High-Fidelity and Diverse Text-to-3D Generation with Variational Score Distillation](https://mlsanity.com/paper/2305.16213/) (2023)
- [One-step Diffusion with Distribution Matching Distillation](https://mlsanity.com/paper/2311.18828/) (2023)
- [Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generation](https://mlsanity.com/paper/2305.01569/) (2023)
- [Human Preference Score v2: A Solid Benchmark for Evaluating Human Preferences of Text-to-Image Synthesis](https://mlsanity.com/paper/2306.09341/) (2023)
- [Inversion-Based Style Transfer with Diffusion Models](https://mlsanity.com/paper/2211.13203/) (2022)
- [Style Injection in Diffusion: A Training-free Approach for Adapting Large-scale Diffusion Models for Style Transfer](https://mlsanity.com/paper/2312.09008/) (2023)
- [Distilling Diffusion Models into Conditional GANs](https://mlsanity.com/paper/2405.05967/) (2024)
- [Aligning Latent and Image Spaces to Connect the Unconnectable](https://mlsanity.com/paper/2104.06954/) (2021)
- [Zero-Shot Contrastive Loss for Text-Guided Diffusion Image Style Transfer](https://mlsanity.com/paper/2303.08622/) (2023)
- [StyleMesh: Style Transfer for Indoor 3D Scene Reconstructions](https://mlsanity.com/paper/2112.01530/) (2021)
- [InstantStyle-Plus: Style Transfer with Content-Preserving in Text-to-Image Generation](https://mlsanity.com/paper/2407.00788/) (2024)
- [Watch Your Steps: Local Image and Scene Editing by Text Instructions](https://mlsanity.com/paper/2308.08947/) (2023)
- [ArtFID: Quantitative Evaluation of Neural Style Transfer](https://mlsanity.com/paper/2207.12280/) (2022)
- [Contrastive Denoising Score for Text-guided Latent Diffusion Image Editing](https://mlsanity.com/paper/2311.18608/) (2023)
- [Rethinking Score Distillation as a Bridge Between Image Distributions](https://mlsanity.com/paper/2406.09417/) (2024)
- [FreeStyle: Free Lunch for Text-guided Style Transfer using Diffusion Models](https://mlsanity.com/paper/2401.15636/) (2024)
- [Magic Insert: Style-Aware Drag-and-Drop](https://mlsanity.com/paper/2407.02489/) (2024)

## Built on by (most cited first, 5 of 5)

- [Personalized Vision via Visual In-Context Learning](https://mlsanity.com/paper/2509.25172/) (2025)
- [LouvreSAE: Sparse Autoencoders for Interpretable and Controllable Style Transfer](https://mlsanity.com/paper/2512.18930/) (2025)
- [Colorful-Noise: Training-Free Low-Frequency Noise Manipulation for Color-Based Conditional Image Generation](https://mlsanity.com/paper/2605.00548/) (2026)
- [Constantly Improving Image Models Need Constantly Improving Benchmarks](https://mlsanity.com/paper/2510.15021/) (2025)
- [Style-Aware Gloss Control for Generative Non-Photorealistic Rendering](https://mlsanity.com/paper/2602.16611/) (2026)

## Abstract (arXiv)

We present Style Matching Score (SMS), a novel optimization method for image stylization with diffusion models. Balancing effective style transfer with content preservation is a long-standing challenge. Unlike existing efforts, our method reframes image stylization as a style distribution matching problem. The target style distribution is estimated from off-the-shelf style-dependent LoRAs via carefully designed score functions. To preserve content information adaptively, we propose Progressive Spectrum Regularization, which operates in the frequency domain to guide stylization progressively from low-frequency layouts to high-frequency details. In addition, we devise a Semantic-Aware Gradient Refinement technique that leverages relevance maps derived from diffusion semantic priors to selectively stylize semantically important regions. The proposed optimization formulation extends stylization from pixel space to parameter space, readily applicable to lightweight feedforward generators for efficient one-step stylization. SMS effectively balances style alignment and content preservation, outperforming state-of-the-art approaches, verified by extensive experiments.