# Overfitting the Data: Compact Neural Video Delivery via Content-aware Feature Modulation

arXiv: https://arxiv.org/abs/2108.08202 · HTML: https://mlsanity.com/paper/2108.08202/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 38 references that are in the map.
- First posted on arXiv 18 Aug 2021.

## Builds on (most cited first, 23 of 23)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://mlsanity.com/paper/1703.03400/) (2017)
- [Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network](https://mlsanity.com/paper/1609.04802/) (2016)
- [Enhanced Deep Residual Networks for Single Image Super-Resolution](https://mlsanity.com/paper/1707.02921/) (2017)
- [Image Super-Resolution Using Very Deep Residual Channel Attention Networks](https://mlsanity.com/paper/1807.02758/) (2018)
- [Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Network](https://mlsanity.com/paper/1609.05158/) (2016)
- [Accurate Image Super-Resolution Using Very Deep Convolutional Networks](https://mlsanity.com/paper/1511.04587/) (2015)
- [Residual Dense Network for Image Super-Resolution](https://mlsanity.com/paper/1802.08797/) (2018)
- [Video Enhancement with Task-Oriented Flow](https://mlsanity.com/paper/1711.09078/) (2017)
- [EDVR: Video Restoration with Enhanced Deformable Convolutional Networks](https://mlsanity.com/paper/1905.02716/) (2019)
- [BasicVSR: The Search for Essential Components in Video Super-Resolution and Beyond](https://mlsanity.com/paper/2012.02181/) (2020)
- [Frame-Recurrent Video Super-Resolution](https://mlsanity.com/paper/1801.04590/) (2018)
- [Real-Time Video Super-Resolution with Spatio-Temporal Networks and Motion Compensation](https://mlsanity.com/paper/1611.05250/) (2016)
- [Fast and Accurate Image Super-Resolution with Deep Laplacian Pyramid Networks](https://mlsanity.com/paper/1710.01992/) (2017)
- [Meta-Transfer Learning for Zero-Shot Super-Resolution](https://mlsanity.com/paper/2002.12213/) (2020)
- [LAPAR: Linearly-Assembled Pixel-Adaptive Regression Network for Single Image Super-Resolution and Beyond](https://mlsanity.com/paper/2105.10422/) (2021)
- [Fast, Accurate and Lightweight Super-Resolution with Neural Architecture Search](https://mlsanity.com/paper/1901.07261/) (2019)
- [Fast Adaptation to Super-Resolution Networks via Meta-Learning](https://mlsanity.com/paper/2001.02905/) (2020)
- [Modulating Image Restoration with Continual Levels via Adaptive Feature Modification Layers](https://mlsanity.com/paper/1904.08118/) (2019)
- [Sensor-Augmented Neural Adaptive Bitrate Video Streaming on UAVs](https://mlsanity.com/paper/1909.10914/) (2019)
- [Neural Enhancement in Content Delivery Systems: The State-of-the-Art and Future Directions](https://mlsanity.com/paper/2010.05838/) (2020)

## Built on by (most cited first, 15 of 15)

- [CaDM: Codec-aware Diffusion Modeling for Neural-enhanced Video Streaming](https://mlsanity.com/paper/2211.08428/) (2022)
- [Instance-Adaptive Video Compression: Improving Neural Codecs by Training on the Test Set](https://mlsanity.com/paper/2111.10302/) (2021)
- [Cloud-Device Collaborative Adaptation to Continual Changing Environments in the Real-world](https://mlsanity.com/paper/2212.00972/) (2022)
- [Learning Mutual Modulation for Self-Supervised Cross-Modal Super-Resolution](https://mlsanity.com/paper/2207.09156/) (2022)
- [Dynamic Low-Rank Instance Adaptation for Universal Neural Image Compression](https://mlsanity.com/paper/2308.07733/) (2023)
- [DiffuseIR:Diffusion Models For Isotropic Reconstruction of 3D Microscopic Images](https://mlsanity.com/paper/2306.12109/) (2023)
- [Efficient Meta-Tuning for Content-aware Neural Video Delivery](https://mlsanity.com/paper/2207.09691/) (2022)
- [Toward Adaptive Semantic Communications: Efficient Data Transmission via Online Learned Nonlinear Transform Source-Channel Coding](https://mlsanity.com/paper/2211.04339/) (2022)
- [Towards High-Quality and Efficient Video Super-Resolution via Spatial-Temporal Data Overfitting](https://mlsanity.com/paper/2303.08331/) (2023)
- [Efficient Deweather Mixture-of-Experts with Uncertainty-aware Feature-wise Linear Modulation](https://mlsanity.com/paper/2312.16610/) (2023)
- [Real-Time Neural-Enhancement for Online Cloud Gaming](https://mlsanity.com/paper/2501.06880/) (2025)
- [Frequency-Aware Re-Parameterization for Over-Fitting Based Image Compression](https://mlsanity.com/paper/2310.08068/) (2023)
- [Data Overfitting for On-Device Super-Resolution with Dynamic Algorithm and Compiler Co-Design](https://mlsanity.com/paper/2407.02813/) (2024)
- [Cell as Point: One-Stage Framework for Efficient Cell Tracking](https://mlsanity.com/paper/2411.14833/) (2024)
- [EPS: Efficient Patch Sampling for Video Overfitting in Deep Super-Resolution Model Training](https://mlsanity.com/paper/2411.16312/) (2024)

## Abstract (arXiv)

Internet video delivery has undergone a tremendous explosion of growth over the past few years. However, the quality of video delivery system greatly depends on the Internet bandwidth. Deep Neural Networks (DNNs) are utilized to improve the quality of video delivery recently. These methods divide a video into chunks, and stream LR video chunks and corresponding content-aware models to the client. The client runs the inference of models to super-resolve the LR chunks. Consequently, a large number of models are streamed in order to deliver a video. In this paper, we first carefully study the relation between models of different chunks, then we tactfully design a joint training framework along with the Content-aware Feature Modulation (CaFM) layer to compress these models for neural video delivery. bf With our method, each video chunk only requires less than 1%  of original parameters to be streamed, achieving even better SR performance. We conduct extensive experiments across various SR backbones, video time length, and scaling factors to demonstrate the advantages of our method. Besides, our method can be also viewed as a new approach of video coding. Our primary experiments achieve better video quality compared with the commercial H.264 and H.265 standard under the same storage cost, showing the great potential of the proposed method. Code is available at:urlhttps://github.com/Neural-video-delivery/CaFM-Pytorch-ICCV2021