# Real-Time Video Super-Resolution with Spatio-Temporal Networks and Motion Compensation

arXiv: https://arxiv.org/abs/1611.05250 · HTML: https://mlsanity.com/paper/1611.05250/

## Key facts

- Cited by 156 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 Nov 2016.

## Built on by (most cited first, 30 of 156)

- [BasicVSR++: Improving Video Super-Resolution with Enhanced Propagation and Alignment](https://mlsanity.com/paper/2104.13371/) (2021)
- [VRT: A Video Restoration Transformer](https://mlsanity.com/paper/2201.12288/) (2022)
- [Recurrent Video Restoration Transformer with Guided Deformable Attention](https://mlsanity.com/paper/2206.02146/) (2022)
- [Video-of-Thought: Step-by-Step Video Reasoning from Perception to Cognition](https://mlsanity.com/paper/2501.03230/) (2024)
- [Efficient Long-Range Attention Network for Image Super-resolution](https://mlsanity.com/paper/2203.06697/) (2022)
- [Video Super-Resolution Transformer](https://mlsanity.com/paper/2106.06847/) (2021)
- [VideoINR: Learning Video Implicit Neural Representation for Continuous Space-Time Super-Resolution](https://mlsanity.com/paper/2206.04647/) (2022)
- [A Simple Baseline for Video Restoration with Grouped Spatial-temporal Shift](https://mlsanity.com/paper/2206.10810/) (2022)
- [Real-World Single Image Super-Resolution: A Brief Review](https://mlsanity.com/paper/2103.02368/) (2021)
- [Learning Trajectory-Aware Transformer for Video Super-Resolution](https://mlsanity.com/paper/2204.04216/) (2022)
- [Motion-Guided Latent Diffusion for Temporally Consistent Real-world Video Super-resolution](https://mlsanity.com/paper/2312.00853/) (2023)
- [STAR: Spatial-Temporal Augmentation with Text-to-Video Models for Real-World Video Super-Resolution](https://mlsanity.com/paper/2501.02976/) (2025)
- [Temporal Modulation Network for Controllable Space-Time Video Super-Resolution](https://mlsanity.com/paper/2104.10642/) (2021)
- [Video Event Restoration Based on Keyframes for Video Anomaly Detection](https://mlsanity.com/paper/2304.05112/) (2023)
- [RSTT: Real-time Spatial Temporal Transformer for Space-Time Video Super-Resolution](https://mlsanity.com/paper/2203.14186/) (2022)
- [Learning Spatial-Temporal Implicit Neural Representations for Event-Guided Video Super-Resolution](https://mlsanity.com/paper/2303.13767/) (2023)
- [Omniscient Video Super-Resolution](https://mlsanity.com/paper/2103.15683/) (2021)
- [VideoGigaGAN: Towards Detail-rich Video Super-Resolution](https://mlsanity.com/paper/2404.12388/) (2024)
- [Mitigating Artifacts in Real-World Video Super-Resolution Models](https://mlsanity.com/paper/2212.07339/) (2022)
- [Video Super-Resolution Transformer with Masked Inter&Intra-Frame Attention](https://mlsanity.com/paper/2401.06312/) (2024)
- [FMA-Net: Flow-Guided Dynamic Filtering and Iterative Feature Refinement with Multi-Attention for Joint Video Super-Resolution and Deblurring](https://mlsanity.com/paper/2401.03707/) (2024)
- [E-CIR: Event-Enhanced Continuous Intensity Recovery](https://mlsanity.com/paper/2203.01935/) (2022)
- [Video Demoireing with Relation-Based Temporal Consistency](https://mlsanity.com/paper/2204.02957/) (2022)
- [A Comprehensive Review of Deep Learning-based Single Image Super-resolution](https://mlsanity.com/paper/2102.09351/) (2021)
- [AnimeSR: Learning Real-World Super-Resolution Models for Animation Videos](https://mlsanity.com/paper/2206.07038/) (2022)
- [Learning Spatial Adaptation and Temporal Coherence in Diffusion Models for Video Super-Resolution](https://mlsanity.com/paper/2403.17000/) (2024)
- [Can SAM Boost Video Super-Resolution?](https://mlsanity.com/paper/2305.06524/) (2023)
- [Reference-based Video Super-Resolution Using Multi-Camera Video Triplets](https://mlsanity.com/paper/2203.14537/) (2022)
- [NTIRE 2022 Challenge on Super-Resolution and Quality Enhancement of Compressed Video: Dataset, Methods and Results](https://mlsanity.com/paper/2204.09314/) (2022)
- [Error Compensation Framework for Flow-Guided Video Inpainting](https://mlsanity.com/paper/2207.10391/) (2022)

## Abstract (arXiv)

Convolutional neural networks have enabled accurate image super-resolution in real-time. However, recent attempts to benefit from temporal correlations in video super-resolution have been limited to naive or inefficient architectures. In this paper, we introduce spatio-temporal sub-pixel convolution networks that effectively exploit temporal redundancies and improve reconstruction accuracy while maintaining real-time speed. Specifically, we discuss the use of early fusion, slow fusion and 3D convolutions for the joint processing of multiple consecutive video frames. We also propose a novel joint motion compensation and video super-resolution algorithm that is orders of magnitude more efficient than competing methods, relying on a fast multi-resolution spatial transformer module that is end-to-end trainable. These contributions provide both higher accuracy and temporally more consistent videos, which we confirm qualitatively and quantitatively. Relative to single-frame models, spatio-temporal networks can either reduce the computational cost by 30% whilst maintaining the same quality or provide a 0.2dB gain for a similar computational cost. Results on publicly available datasets demonstrate that the proposed algorithms surpass current state-of-the-art performance in both accuracy and efficiency.