# Style-Aware Normalized Loss for Improving Arbitrary Style Transfer

arXiv: https://arxiv.org/abs/2104.10064 · HTML: https://mlsanity.com/paper/2104.10064/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- Builds on 40 of its 56 references that are in the map.
- First posted on arXiv 18 Apr 2021.

## Builds on (most cited first, 30 of 40)

- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Perceptual Losses for Real-Time Style Transfer and Super-Resolution](https://mlsanity.com/paper/1603.08155/) (2016)
- [Describing Textures in the Wild](https://mlsanity.com/paper/1311.3618/) (2013)
- [Arbitrary Style Transfer in Real-time with Adaptive Instance Normalization](https://mlsanity.com/paper/1703.06868/) (2017)
- [Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semantics](https://mlsanity.com/paper/1705.07115/) (2017)
- [GradNorm: Gradient Normalization for Adaptive Loss Balancing in Deep Multitask Networks](https://mlsanity.com/paper/1711.02257/) (2017)
- [End-to-End Multi-Task Learning with Attention](https://mlsanity.com/paper/1803.10704/) (2018)
- [Universal Style Transfer via Feature Transforms](https://mlsanity.com/paper/1705.08086/) (2017)
- [A Learned Representation For Artistic Style](https://mlsanity.com/paper/1610.07629/) (2016)
- [Improved Texture Networks: Maximizing Quality and Diversity in Feed-forward Stylization and Texture Synthesis](https://mlsanity.com/paper/1701.02096/) (2017)
- [Arbitrary Style Transfer with Style-Attentional Networks](https://mlsanity.com/paper/1812.02342/) (2018)
- [Precomputed Real-Time Texture Synthesis with Markovian Generative Adversarial Networks](https://mlsanity.com/paper/1604.04382/) (2016)
- [Texture Networks: Feed-forward Synthesis of Textures and Stylized Images](https://mlsanity.com/paper/1603.03417/) (2016)
- [Deep Photo Style Transfer](https://mlsanity.com/paper/1703.07511/) (2017)
- [Style Transfer by Relaxed Optimal Transport and Self-Similarity](https://mlsanity.com/paper/1904.12785/) (2019)
- [Photorealistic Style Transfer via Wavelet Transforms](https://mlsanity.com/paper/1903.09760/) (2019)
- [Demystifying Neural Style Transfer](https://mlsanity.com/paper/1701.01036/) (2017)
- [Avatar-Net: Multi-scale Zero-shot Style Transfer by Feature Decoration](https://mlsanity.com/paper/1805.03857/) (2018)
- [Coherent Online Video Style Transfer](https://mlsanity.com/paper/1703.09211/) (2017)
- [Fast Patch-based Style Transfer of Arbitrary Style](https://mlsanity.com/paper/1612.04337/) (2016)
- [StyleBank: An Explicit Representation for Neural Image Style Transfer](https://mlsanity.com/paper/1703.09210/) (2017)
- [Exploring the structure of a real-time, arbitrary neural artistic stylization network](https://mlsanity.com/paper/1705.06830/) (2017)
- [Dynamic Instance Normalization for Arbitrary Style Transfer](https://mlsanity.com/paper/1911.06953/) (2019)
- [Arbitrary Style Transfer with Deep Feature Reshuffle](https://mlsanity.com/paper/1805.04103/) (2018)
- [Stable and Controllable Neural Texture Synthesis and Style Transfer Using Histogram Losses](https://mlsanity.com/paper/1701.08893/) (2017)
- [Multi-style Generative Network for Real-time Transfer](https://mlsanity.com/paper/1703.06953/) (2017)
- [Diversified Texture Synthesis with Feed-forward Networks](https://mlsanity.com/paper/1703.01664/) (2017)
- [A Style-Aware Content Loss for Real-time HD Style Transfer](https://mlsanity.com/paper/1807.10201/) (2018)
- [Collaborative Distillation for Ultra-Resolution Universal Style Transfer](https://mlsanity.com/paper/2003.08436/) (2020)

## Built on by (most cited first, 30 of 30)

- [StyleDiffusion: Controllable Disentangled Style Transfer via Diffusion Models](https://mlsanity.com/paper/2308.07863/) (2023)
- [Domain Enhanced Arbitrary Image Style Transfer via Contrastive Learning](https://mlsanity.com/paper/2205.09542/) (2022)
- [CCPL: Contrastive Coherence Preserving Loss for Versatile Style Transfer](https://mlsanity.com/paper/2207.04808/) (2022)
- [Domain-Aware Universal Style Transfer](https://mlsanity.com/paper/2108.04441/) (2021)
- [Cross-Domain Correlation Distillation for Unsupervised Domain Adaptation in Nighttime Semantic Segmentation](https://mlsanity.com/paper/2205.00858/) (2022)
- [Wavelet Knowledge Distillation: Towards Efficient Image-to-Image Translation](https://mlsanity.com/paper/2203.06321/) (2022)
- [Neural Preset for Color Style Transfer](https://mlsanity.com/paper/2303.13511/) (2023)
- [AesPA-Net: Aesthetic Pattern-Aware Style Transfer Networks](https://mlsanity.com/paper/2307.09724/) (2023)
- [Unsupervised Monocular Depth Estimation in Highly Complex Environments](https://mlsanity.com/paper/2107.13137/) (2021)
- [AesUST: Towards Aesthetic-Enhanced Universal Style Transfer](https://mlsanity.com/paper/2208.13016/) (2022)
- [MicroAST: Towards Super-Fast Ultra-Resolution Arbitrary Style Transfer](https://mlsanity.com/paper/2211.15313/) (2022)
- [All-to-key Attention for Arbitrary Style Transfer](https://mlsanity.com/paper/2212.04105/) (2022)
- [Texture Reformer: Towards Fast and Universal Interactive Texture Transfer](https://mlsanity.com/paper/2112.02788/) (2021)
- [Learning Dynamic Style Kernels for Artistic Style Transfer](https://mlsanity.com/paper/2304.00414/) (2023)
- [Region-aware Knowledge Distillation for Efficient Image-to-Image Translation](https://mlsanity.com/paper/2205.12451/) (2022)
- [StylerDALLE: Language-Guided Style Transfer Using a Vector-Quantized Tokenizer of a Large-Scale Generative Model](https://mlsanity.com/paper/2303.09268/) (2023)
- [Industrial Style Transfer with Large-scale Geometric Warping and Content Preservation](https://mlsanity.com/paper/2203.12835/) (2022)
- [Video Color Grading via Look-Up Table Generation](https://mlsanity.com/paper/2508.00548/) (2025)
- [Image Harmonization by Matching Regional References](https://mlsanity.com/paper/2204.04715/) (2022)
- [Quality Evaluation of Arbitrary Style Transfer: Subjective Study and Objective Metric](https://mlsanity.com/paper/2208.00623/) (2022)
- [Causal-DFQ: Causality Guided Data-free Network Quantization](https://mlsanity.com/paper/2309.13682/) (2023)
- [ArtFusion: Controllable Arbitrary Style Transfer using Dual Conditional Latent Diffusion Models](https://mlsanity.com/paper/2306.09330/) (2023)
- [SpectralCLIP: Preventing Artifacts in Text-Guided Style Transfer from a Spectral Perspective](https://mlsanity.com/paper/2303.09270/) (2023)
- [Interactive Style Transfer: All is Your Palette](https://mlsanity.com/paper/2203.13470/) (2022)
- [Controlling Neural Style Transfer with Deep Reinforcement Learning](https://mlsanity.com/paper/2310.00405/) (2023)
- [Towards Generalized Multimodal Homography Estimation](https://mlsanity.com/paper/2603.03956/) (2026)
- [Style-Guided Inference of Transformer for High-resolution Image Synthesis](https://mlsanity.com/paper/2210.05533/) (2022)
- [Line Search-Based Feature Transformation for Fast, Stable, and Tunable Content-Style Control in Photorealistic Style Transfer](https://mlsanity.com/paper/2210.05996/) (2022)
- [Retinex-guided Channel-grouping based Patch Swap for Arbitrary Style Transfer](https://mlsanity.com/paper/2309.10528/) (2023)
- [RL-I2IT: Image-to-Image Translation with Deep Reinforcement Learning](https://mlsanity.com/paper/2309.13672/) (2023)

## Abstract (arXiv)

Neural Style Transfer (NST) has quickly evolved from single-style to infinite-style models, also known as Arbitrary Style Transfer (AST). Although appealing results have been widely reported in literature, our empirical studies on four well-known AST approaches (GoogleMagenta, AdaIN, LinearTransfer, and SANet) show that more than 50% of the time, AST stylized images are not acceptable to human users, typically due to under- or over-stylization. We systematically study the cause of this imbalanced style transferability (IST) and propose a simple yet effective solution to mitigate this issue. Our studies show that the IST issue is related to the conventional AST style loss, and reveal that the root cause is the equal weightage of training samples irrespective of the properties of their corresponding style images, which biases the model towards certain styles. Through investigation of the theoretical bounds of the AST style loss, we propose a new loss that largely overcomes IST. Theoretical analysis and experimental results validate the effectiveness of our loss, with over 80% relative improvement in style deception rate and 98% relatively higher preference in human evaluation.