# EPS: Efficient Patch Sampling for Video Overfitting in Deep Super-Resolution Model Training

arXiv: https://arxiv.org/abs/2411.16312 · HTML: https://mlsanity.com/paper/2411.16312/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 30 of its 53 references that are in the map.
- First posted on arXiv 25 Nov 2024.

## Builds on (most cited first, 30 of 30)

- [SwinIR: Image Restoration Using Swin Transformer](https://mlsanity.com/paper/2108.10257/) (2021)
- [Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Network](https://mlsanity.com/paper/1609.05158/) (2016)
- [FlowNet: Learning Optical Flow with Convolutional Networks](https://mlsanity.com/paper/1504.06852/) (2015)
- [An Empirical Study of Example Forgetting during Deep Neural Network Learning](https://mlsanity.com/paper/1812.05159/) (2018)
- [Accelerating the Super-Resolution Convolutional Neural Network](https://mlsanity.com/paper/1608.00367/) (2016)
- [Fast, Accurate, and Lightweight Super-Resolution with Cascading Residual Network](https://mlsanity.com/paper/1803.08664/) (2018)
- [Dataset Distillation](https://mlsanity.com/paper/1811.10959/) (2018)
- [Video Compression through Image Interpolation](https://mlsanity.com/paper/1804.06919/) (2018)
- [Video Compression With Rate-Distortion Autoencoders](https://mlsanity.com/paper/1908.05717/) (2019)
- [Learned Video Compression](https://mlsanity.com/paper/1811.06981/) (2018)
- [Image and Video Compression with Neural Networks: A Review](https://mlsanity.com/paper/1904.03567/) (2019)
- [ClassSR: A General Framework to Accelerate Super-Resolution Networks by Data Characteristic](https://mlsanity.com/paper/2103.04039/) (2021)
- [MEST: Accurate and Fast Memory-Economic Sparse Training Framework on the Edge](https://mlsanity.com/paper/2110.14032/) (2021)
- [Content Adaptive and Error Propagation Aware Deep Video Compression](https://mlsanity.com/paper/2003.11282/) (2020)
- [Wide Activation for Efficient and Accurate Image Super-Resolution](https://mlsanity.com/paper/1808.08718/) (2018)
- [Frequency-Assisted Mamba for Remote Sensing Image Super-Resolution](https://mlsanity.com/paper/2405.04964/) (2024)
- [AdaPool: Exponential Adaptive Pooling for Information-Retaining Downsampling](https://mlsanity.com/paper/2111.00772/) (2021)
- [Omniscient Video Super-Resolution](https://mlsanity.com/paper/2103.15683/) (2021)
- [CAMixerSR: Only Details Need More "Attention"](https://mlsanity.com/paper/2402.19289/) (2024)
- [GALAXY: Graph-based Active Learning at the Extreme](https://mlsanity.com/paper/2202.01402/) (2022)
- [Efficient Video Compression via Content-Adaptive Super-Resolution](https://mlsanity.com/paper/2104.02322/) (2021)
- [Local-Global Temporal Difference Learning for Satellite Video Super-Resolution](https://mlsanity.com/paper/2304.04421/) (2023)
- [Temporal Consistency Learning of inter-frames for Video Super-Resolution](https://mlsanity.com/paper/2211.01639/) (2022)
- [Overfitting the Data: Compact Neural Video Delivery via Content-aware Feature Modulation](https://mlsanity.com/paper/2108.08202/) (2021)
- [Iterative Patch Selection for High-Resolution Image Recognition](https://mlsanity.com/paper/2210.13007/) (2022)
- [Convolutional Neural Network-Based Block Up-sampling for Intra Frame Coding](https://mlsanity.com/paper/1702.06728/) (2017)
- [Efficient Meta-Tuning for Content-aware Neural Video Delivery](https://mlsanity.com/paper/2207.09691/) (2022)
- [Towards High-Quality and Efficient Video Super-Resolution via Spatial-Temporal Data Overfitting](https://mlsanity.com/paper/2303.08331/) (2023)
- [SamplingAug: On the Importance of Patch Sampling Augmentation for Single Image Super-Resolution](https://mlsanity.com/paper/2111.15185/) (2021)
- [A Survey on Energy Consumption and Environmental Impact of Video Streaming](https://mlsanity.com/paper/2401.09854/) (2024)

## Abstract (arXiv)

Leveraging the overfitting property of deep neural networks (DNNs) is trending in video delivery systems to enhance video quality within bandwidth limits. Existing approaches transmit overfitted super-resolution (SR) model streams for low-resolution (LR) bitstreams, which are used to reconstruct high-resolution (HR) videos at the decoder. Although these approaches show promising results, the huge computational costs of training a large number of video frames limit their practical applications. To overcome this challenge, we propose an efficient patch sampling method named EPS for video SR network overfitting, which identifies the most valuable training patches from video frames. To this end, we first present two low-complexity Discrete Cosine Transform (DCT)-based spatial-temporal features to measure the complexity score of each patch directly. By analyzing the histogram distribution of these features, we then categorize all possible patches into different clusters and select training patches from the cluster with the highest spatial-temporal information. The number of sampled patches is adaptive based on the video content, addressing the trade-off between training complexity and efficiency. Our method reduces the number of training patches by 75.00% to 91.69%, depending on the resolution and number of clusters, while preserving high video quality and greatly improving training efficiency. Our method speeds up patch sampling by up to 82.1x compared to the state-of-the-art patch sampling technique (EMT).