# Frame Difference-Based Temporal Loss for Video Stylization

arXiv: https://arxiv.org/abs/2102.05822 · HTML: https://mlsanity.com/paper/2102.05822/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 41 references that are in the map.
- First posted on arXiv 11 Feb 2021.

## Builds on (most cited first, 17 of 17)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Perceptual Losses for Real-Time Style Transfer and Super-Resolution](https://mlsanity.com/paper/1603.08155/) (2016)
- [Arbitrary Style Transfer in Real-time with Adaptive Instance Normalization](https://mlsanity.com/paper/1703.06868/) (2017)
- [FlowNet: Learning Optical Flow with Convolutional Networks](https://mlsanity.com/paper/1504.06852/) (2015)
- [FlowNet 2.0: Evolution of Optical Flow Estimation with Deep Networks](https://mlsanity.com/paper/1612.01925/) (2016)
- [A Learned Representation For Artistic Style](https://mlsanity.com/paper/1610.07629/) (2016)
- [Learning Blind Video Temporal Consistency](https://mlsanity.com/paper/1808.00449/) (2018)
- [Texture Networks: Feed-forward Synthesis of Textures and Stylized Images](https://mlsanity.com/paper/1603.03417/) (2016)
- [Coherent Online Video Style Transfer](https://mlsanity.com/paper/1703.09211/) (2017)
- [StyleBank: An Explicit Representation for Neural Image Style Transfer](https://mlsanity.com/paper/1703.09210/) (2017)
- [Diversified Texture Synthesis with Feed-forward Networks](https://mlsanity.com/paper/1703.01664/) (2017)
- [Artistic style transfer for videos and spherical images](https://mlsanity.com/paper/1708.04538/) (2017)
- [Characterizing and Improving Stability in Neural Style Transfer](https://mlsanity.com/paper/1705.02092/) (2017)
- [A Content Transformation Block For Image Style Transfer](https://mlsanity.com/paper/2003.08407/) (2020)

## Built on by (most cited first, 3 of 3)

- [Personalized Face Inpainting with Diffusion Models by Parallel Visual Attention](https://mlsanity.com/paper/2312.03556/) (2023)
- [Video Color Grading via Look-Up Table Generation](https://mlsanity.com/paper/2508.00548/) (2025)
- [Expressive Talking Head Video Encoding in StyleGAN2 Latent-Space](https://mlsanity.com/paper/2203.14512/) (2022)

## Abstract (arXiv)

Neural style transfer models have been used to stylize an ordinary video to specific styles. To ensure temporal inconsistency between the frames of the stylized video, a common approach is to estimate the optic flow of the pixels in the original video and make the generated pixels match the estimated optical flow. This is achieved by minimizing an optical flow-based (OFB) loss during model training. However, optical flow estimation is itself a challenging task, particularly in complex scenes. In addition, it incurs a high computational cost. We propose a much simpler temporal loss called the frame difference-based (FDB) loss to solve the temporal inconsistency problem. It is defined as the distance between the difference between the stylized frames and the difference between the original frames. The differences between the two frames are measured in both the pixel space and the feature space specified by the convolutional neural networks. A set of human behavior experiments involving 62 subjects with 25,600 votes showed that the performance of the proposed FDB loss matched that of the OFB loss. The performance was measured by subjective evaluation of stability and stylization quality of the generated videos on two typical video stylization models. The results suggest that the proposed FDB loss is a strong alternative to the commonly used OFB loss for video stylization.