# Efficient Space-time Video Super Resolution using Low-Resolution Flow and Mask Upsampling

arXiv: https://arxiv.org/abs/2104.05778 · HTML: https://mlsanity.com/paper/2104.05778/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 41 references that are in the map.
- First posted on arXiv 12 Apr 2021.

## Builds on (most cited first, 30 of 32)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Enhanced Deep Residual Networks for Single Image Super-Resolution](https://mlsanity.com/paper/1707.02921/) (2017)
- [UNet++: A Nested U-Net Architecture for Medical Image Segmentation](https://mlsanity.com/paper/1807.10165/) (2018)
- [Image Super-Resolution Using Deep Convolutional Networks](https://mlsanity.com/paper/1501.00092/) (2014)
- [Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Network](https://mlsanity.com/paper/1609.05158/) (2016)
- [PWC-Net: CNNs for Optical Flow Using Pyramid, Warping, and Cost Volume](https://mlsanity.com/paper/1709.02371/) (2017)
- [Video Enhancement with Task-Oriented Flow](https://mlsanity.com/paper/1711.09078/) (2017)
- [EDVR: Video Restoration with Enhanced Deformable Convolutional Networks](https://mlsanity.com/paper/1905.02716/) (2019)
- [Super SloMo: High Quality Estimation of Multiple Intermediate Frames for Video Interpolation](https://mlsanity.com/paper/1712.00080/) (2017)
- [Depth-Aware Video Frame Interpolation](https://mlsanity.com/paper/1904.00830/) (2019)
- [TDAN: Temporally Deformable Alignment Network for Video Super-Resolution](https://mlsanity.com/paper/1812.02898/) (2018)
- [Video Frame Interpolation via Adaptive Separable Convolution](https://mlsanity.com/paper/1708.01692/) (2017)
- [Video Frame Synthesis using Deep Voxel Flow](https://mlsanity.com/paper/1702.02463/) (2017)
- [Video Frame Interpolation via Adaptive Convolution](https://mlsanity.com/paper/1703.07514/) (2017)
- [Real-Time Video Super-Resolution with Spatio-Temporal Networks and Motion Compensation](https://mlsanity.com/paper/1611.05250/) (2016)
- [Context-aware Synthesis for Video Frame Interpolation](https://mlsanity.com/paper/1803.10967/) (2018)
- [Detail-revealing Deep Video Super-resolution](https://mlsanity.com/paper/1704.02738/) (2017)
- [Recurrent Back-Projection Network for Video Super-Resolution](https://mlsanity.com/paper/1903.10128/) (2019)
- [Quadratic video interpolation](https://mlsanity.com/paper/1911.00627/) (2019)
- [Image Super-Resolution by Neural Texture Transfer](https://mlsanity.com/paper/1903.00834/) (2019)
- [Video Super-Resolution with Recurrent Structure-Detail Network](https://mlsanity.com/paper/2008.00455/) (2020)
- [MEMC-Net: Motion Estimation and Motion Compensation Driven Neural Network for Video Interpolation and Enhancement](https://mlsanity.com/paper/1810.08768/) (2018)
- [FLAVR: Flow-Agnostic Video Representations for Fast Frame Interpolation](https://mlsanity.com/paper/2012.08512/) (2020)
- [Zooming Slow-Mo: Fast and Accurate One-Stage Space-Time Video Super-Resolution](https://mlsanity.com/paper/2002.11616/) (2020)
- [Residual Conv-Deconv Grid Network for Semantic Segmentation](https://mlsanity.com/paper/1707.07958/) (2017)
- [Space-Time-Aware Multi-Resolution Video Enhancement](https://mlsanity.com/paper/2003.13170/) (2020)
- [Learning Image Matching by Simply Watching Video](https://mlsanity.com/paper/1603.06041/) (2016)
- [Fast Spatio-Temporal Residual Network for Video Super-Resolution](https://mlsanity.com/paper/1904.02870/) (2019)
- [FISR: Deep Joint Frame Interpolation and Super-Resolution with a Multi-scale Temporal Loss](https://mlsanity.com/paper/1912.07213/) (2019)

## Built on by (most cited first, 9 of 9)

- [DeMFI: Deep Joint Deblurring and Multi-Frame Interpolation with Flow-Guided Attentive Correlation and Recursive Boosting](https://mlsanity.com/paper/2111.09985/) (2021)
- [NTIRE 2021 Challenge on Video Super-Resolution](https://mlsanity.com/paper/2104.14852/) (2021)
- [HR-INR: Continuous Space-Time Video Super-Resolution via Event Camera](https://mlsanity.com/paper/2405.13389/) (2024)
- [A Survey of Deep Learning Video Super-Resolution](https://mlsanity.com/paper/2506.03216/) (2025)
- [Non-linear Motion Estimation for Video Frame Interpolation using Space-time Convolutions](https://mlsanity.com/paper/2201.11407/) (2022)
- [You Only Align Once: Bidirectional Interaction for Spatial-Temporal Video Super-Resolution](https://mlsanity.com/paper/2207.06345/) (2022)
- [Learning Spatio-Temporal Downsampling for Effective Video Upscaling](https://mlsanity.com/paper/2203.08140/) (2022)
- [Global Spatial-Temporal Information-based Residual ConvLSTM for Video Space-Time Super-Resolution](https://mlsanity.com/paper/2407.08466/) (2024)
- [NNVISR: Bring Neural Network Video Interpolation and Super Resolution into Video Processing Framework](https://mlsanity.com/paper/2308.03121/) (2023)

## Abstract (arXiv)

This paper explores an efficient solution for Space-time Super-Resolution, aiming to generate High-resolution Slow-motion videos from Low Resolution and Low Frame rate videos. A simplistic solution is the sequential running of Video Super Resolution and Video Frame interpolation models. However, this type of solutions are memory inefficient, have high inference time, and could not make the proper use of space-time relation property. To this extent, we first interpolate in LR space using quadratic modeling. Input LR frames are super-resolved using a state-of-the-art Video Super-Resolution method. Flowmaps and blending mask which are used to synthesize LR interpolated frame is reused in HR space using bilinear upsampling. This leads to a coarse estimate of HR intermediate frame which often contains artifacts along motion boundaries. We use a refinement network to improve the quality of HR intermediate frame via residual learning. Our model is lightweight and performs better than current state-of-the-art models in REDS STSR Validation set.