# PCA-Based Knowledge Distillation Towards Lightweight and Content-Style Balanced Photorealistic Style Transfer Models

arXiv: https://arxiv.org/abs/2203.13452 · HTML: https://mlsanity.com/paper/2203.13452/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Placed in Multimodal distillation, one of 12 directions in Knowledge distillation.
- Builds on 30 of its 56 references that are in the map.
- First posted on arXiv 25 Mar 2022.

## Placed in

- Knowledge distillation / [Multimodal distillation](https://mlsanity.com/distill/multimodal-distillation/)

## Builds on (most cited first, 30 of 30)

- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications](https://mlsanity.com/paper/1704.04861/) (2017)
- [Perceptual Losses for Real-Time Style Transfer and Super-Resolution](https://mlsanity.com/paper/1603.08155/) (2016)
- [Arbitrary Style Transfer in Real-time with Adaptive Instance Normalization](https://mlsanity.com/paper/1703.06868/) (2017)
- [Relational Knowledge Distillation](https://mlsanity.com/paper/1904.05068/) (2019)
- [Improved Knowledge Distillation via Teacher Assistant](https://mlsanity.com/paper/1902.03393/) (2019)
- [Similarity-Preserving Knowledge Distillation](https://mlsanity.com/paper/1907.09682/) (2019)
- [A Neural Algorithm of Artistic Style](https://mlsanity.com/paper/1508.06576/) (2015)
- [NIMA: Neural Image Assessment](https://mlsanity.com/paper/1709.05424/) (2017)
- [On the Efficacy of Knowledge Distillation](https://mlsanity.com/paper/1910.01348/) (2019)
- [Universal Style Transfer via Feature Transforms](https://mlsanity.com/paper/1705.08086/) (2017)
- [Correlation Congruence for Knowledge Distillation](https://mlsanity.com/paper/1904.01802/) (2019)
- [Structured Knowledge Distillation for Dense Prediction](https://mlsanity.com/paper/1903.04197/) (2019)
- [Deep Photo Style Transfer](https://mlsanity.com/paper/1703.07511/) (2017)
- [Photorealistic Style Transfer via Wavelet Transforms](https://mlsanity.com/paper/1903.09760/) (2019)
- [Towards Understanding Knowledge Distillation](https://mlsanity.com/paper/2105.13093/) (2021)
- [Demystifying Neural Style Transfer](https://mlsanity.com/paper/1701.01036/) (2017)
- [A Closed-form Solution to Photorealistic Image Stylization](https://mlsanity.com/paper/1802.06474/) (2018)
- [Avatar-Net: Multi-scale Zero-shot Style Transfer by Feature Decoration](https://mlsanity.com/paper/1805.03857/) (2018)
- [Apprentice: Using Knowledge Distillation Techniques To Improve Low-Precision Network Accuracy](https://mlsanity.com/paper/1711.05852/) (2017)
- [Fast Patch-based Style Transfer of Arbitrary Style](https://mlsanity.com/paper/1612.04337/) (2016)
- [Exploring the structure of a real-time, arbitrary neural artistic stylization network](https://mlsanity.com/paper/1705.06830/) (2017)
- [Collaborative Distillation for Ultra-Resolution Universal Style Transfer](https://mlsanity.com/paper/2003.08436/) (2020)
- [Ultrafast Photorealistic Style Transfer via Neural Architecture Search](https://mlsanity.com/paper/1912.02398/) (2019)
- [Optical Flow Distillation: Towards Efficient and Stable Video Style Transfer](https://mlsanity.com/paper/2007.05146/) (2020)
- [Joint Bilateral Learning for Real-time Universal Photorealistic Style Transfer](https://mlsanity.com/paper/2004.10955/) (2020)
- [PhotoWCT2: Compact Autoencoder for Photorealistic Style Transfer Resulting from Blockwise Training and Skip Connections of High-Frequency Residuals](https://mlsanity.com/paper/2110.11995/) (2021)
- [Interpretable Embedding Procedure Knowledge Transfer via Stacked Principal Component Analysis and Graph Neural Network](https://mlsanity.com/paper/2104.13561/) (2021)

## Built on by (most cited first, 11 of 11)

- [ArtFID: Quantitative Evaluation of Neural Style Transfer](https://mlsanity.com/paper/2207.12280/) (2022)
- [CAP-VSTNet: Content Affinity Preserved Versatile Style Transfer](https://mlsanity.com/paper/2303.17867/) (2023)
- [NLUT: Neural-based 3D Lookup Tables for Video Photorealistic Style Transfer](https://mlsanity.com/paper/2303.09170/) (2023)
- [Video Color Grading via Look-Up Table Generation](https://mlsanity.com/paper/2508.00548/) (2025)
- [Modernizing Old Photos Using Multiple References via Photorealistic Style Transfer](https://mlsanity.com/paper/2304.04461/) (2023)
- [EEG-Driven 3D Object Reconstruction with Style Consistency and Diffusion Prior](https://mlsanity.com/paper/2410.20981/) (2024)
- [NCST: Neural-based Color Style Transfer for Video Retouching](https://mlsanity.com/paper/2411.00335/) (2024)
- [Line Search-Based Feature Transformation for Fast, Stable, and Tunable Content-Style Control in Photorealistic Style Transfer](https://mlsanity.com/paper/2210.05996/) (2022)
- [MRStyle: A Unified Framework for Color Style Transfer with Multi-Modality Reference](https://mlsanity.com/paper/2409.05250/) (2024)
- [Pluggable Style Representation Learning for Multi-Style Transfer](https://mlsanity.com/paper/2503.20368/) (2025)
- [Edge-Aware Image Manipulation via Diffusion Models with a Novel Structure-Preservation Loss](https://mlsanity.com/paper/2601.16645/) (2026)

## Abstract (arXiv)

Photorealistic style transfer entails transferring the style of a reference image to another image so the result seems like a plausible photo. Our work is inspired by the observation that existing models are slow due to their large sizes. We introduce PCA-based knowledge distillation to distill lightweight models and show it is motivated by theory. To our knowledge, this is the first knowledge distillation method for photorealistic style transfer. Our experiments demonstrate its versatility for use with different backbone architectures, VGG and MobileNet, across six image resolutions. Compared to existing models, our top-performing model runs at speeds 5-20x faster using at most 1% of the parameters. Additionally, our distilled models achieve a better balance between stylization strength and content preservation than existing models. To support reproducing our method and models, we share the code at textithttps://github.com/chiutaiyin/PCA-Knowledge-Distillation.