# Collaborative Distillation for Ultra-Resolution Universal Style Transfer

arXiv: https://arxiv.org/abs/2003.08436 · HTML: https://mlsanity.com/paper/2003.08436/

## Key facts

- Cited by 52 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 18 Mar 2020.

## Built on by (most cited first, 30 of 52)

- [LightGaussian: Unbounded 3D Gaussian Compression with 15x Reduction and 200+ FPS](https://mlsanity.com/paper/2311.17245/) (2023)
- [Style Injection in Diffusion: A Training-free Approach for Adapting Large-scale Diffusion Models for Style Transfer](https://mlsanity.com/paper/2312.09008/) (2023)
- [StyTr2: Image Style Transfer with Transformers](https://mlsanity.com/paper/2105.14576/) (2021)
- [ArtFlow: Unbiased Image Style Transfer via Reversible Neural Flows](https://mlsanity.com/paper/2103.16877/) (2021)
- [R2L: Distilling Neural Radiance Field to Neural Light Field for Efficient Novel View Synthesis](https://mlsanity.com/paper/2203.17261/) (2022)
- [Learning Student-Friendly Teacher Networks for Knowledge Distillation](https://mlsanity.com/paper/2102.07650/) (2021)
- [ArtFID: Quantitative Evaluation of Neural Style Transfer](https://mlsanity.com/paper/2207.12280/) (2022)
- [CCPL: Contrastive Coherence Preserving Loss for Versatile Style Transfer](https://mlsanity.com/paper/2207.04808/) (2022)
- [DiffStyler: Controllable Dual Diffusion for Text-Driven Image Stylization](https://mlsanity.com/paper/2211.10682/) (2022)
- [Rethinking Style Transfer: From Pixels to Parameterized Brushstrokes](https://mlsanity.com/paper/2103.17185/) (2021)
- [Domain-Aware Universal Style Transfer](https://mlsanity.com/paper/2108.04441/) (2021)
- [Style-Aware Normalized Loss for Improving Arbitrary Style Transfer](https://mlsanity.com/paper/2104.10064/) (2021)
- [EleGANt: Exquisite and Locally Editable GAN for Makeup Transfer](https://mlsanity.com/paper/2207.09840/) (2022)
- [CAP-VSTNet: Content Affinity Preserved Versatile Style Transfer](https://mlsanity.com/paper/2303.17867/) (2023)
- [DyLiN: Making Light Field Networks Dynamic](https://mlsanity.com/paper/2303.14243/) (2023)
- [SGSST: Scaling Gaussian Splatting StyleTransfer](https://mlsanity.com/paper/2412.03371/) (2024)
- [MicroAST: Towards Super-Fast Ultra-Resolution Arbitrary Style Transfer](https://mlsanity.com/paper/2211.15313/) (2022)
- [Facial Attribute Transformers for Precise and Robust Makeup Transfer](https://mlsanity.com/paper/2104.02894/) (2021)
- [InstaStyle: Inversion Noise of a Stylized Image is Secretly a Style Adviser](https://mlsanity.com/paper/2311.15040/) (2023)
- [Z^*: Zero-shot Style Transfer via Attention Rearrangement](https://mlsanity.com/paper/2311.16491/) (2023)
- [Towards Ultra-Resolution Neural Style Transfer via Thumbnail Instance Normalization](https://mlsanity.com/paper/2103.11784/) (2021)
- [Name Your Style: An Arbitrary Artist-aware Image Style Transfer](https://mlsanity.com/paper/2202.13562/) (2022)
- [S2WAT: Image Style Transfer via Hierarchical Vision Transformer using Strips Window Attention](https://mlsanity.com/paper/2210.12381/) (2022)
- [DASK: Distribution Rehearsing via Adaptive Style Kernel Learning for Exemplar-Free Lifelong Person Re-Identification](https://mlsanity.com/paper/2412.09224/) (2024)
- [PCA-Based Knowledge Distillation Towards Lightweight and Content-Style Balanced Photorealistic Style Transfer Models](https://mlsanity.com/paper/2203.13452/) (2022)
- [Industrial Style Transfer with Large-scale Geometric Warping and Content Preservation](https://mlsanity.com/paper/2203.12835/) (2022)
- [Distilling Out-of-Distribution Robustness from Vision-Language Foundation Models](https://mlsanity.com/paper/2311.01441/) (2023)
- [Cosine Similarity Knowledge Distillation for Individual Class Information Transfer](https://mlsanity.com/paper/2311.14307/) (2023)
- [ArtFusion: Controllable Arbitrary Style Transfer using Dual Conditional Latent Diffusion Models](https://mlsanity.com/paper/2306.09330/) (2023)
- [Robustness-Reinforced Knowledge Distillation with Correlation Distance and Network Pruning](https://mlsanity.com/paper/2311.13934/) (2023)

## Abstract (arXiv)

Universal style transfer methods typically leverage rich representations from deep Convolutional Neural Network (CNN) models (e.g., VGG-19) pre-trained on large collections of images. Despite the effectiveness, its application is heavily constrained by the large model size to handle ultra-resolution images given limited memory. In this work, we present a new knowledge distillation method (named Collaborative Distillation) for encoder-decoder based neural style transfer to reduce the convolutional filters. The main idea is underpinned by a finding that the encoder-decoder pairs construct an exclusive collaborative relationship, which is regarded as a new kind of knowledge for style transfer models. Moreover, to overcome the feature size mismatch when applying collaborative distillation, a linear embedding loss is introduced to drive the student network to learn a linear embedding of the teacher's features. Extensive experiments show the effectiveness of our method when applied to different universal style transfer approaches (WCT and AdaIN), even if the model size is reduced by 15.5 times. Especially, on WCT with the compressed models, we achieve ultra-resolution (over 40 megapixels) universal style transfer on a 12GB GPU for the first time. Further experiments on optimization-based stylization scheme show the generality of our algorithm on different stylization paradigms. Our code and trained models are available at https://github.com/mingsun-tse/collaborative-distillation.