# Continuous Space-Time Video Super-Resolution Utilizing Long-Range Temporal Information

arXiv: https://arxiv.org/abs/2302.13256 · HTML: https://mlsanity.com/paper/2302.13256/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 39 of its 50 references that are in the map.
- First posted on arXiv 26 Feb 2023.

## Builds on (most cited first, 30 of 39)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting](https://mlsanity.com/paper/1506.04214/) (2015)
- [Image Super-Resolution Using Very Deep Residual Channel Attention Networks](https://mlsanity.com/paper/1807.02758/) (2018)
- [Video Enhancement with Task-Oriented Flow](https://mlsanity.com/paper/1711.09078/) (2017)
- [Learning Continuous Image Representation with Local Implicit Image Function](https://mlsanity.com/paper/2012.09161/) (2020)
- [EDVR: Video Restoration with Enhanced Deformable Convolutional Networks](https://mlsanity.com/paper/1905.02716/) (2019)
- [Deep Laplacian Pyramid Networks for Fast and Accurate Super-Resolution](https://mlsanity.com/paper/1704.03915/) (2017)
- [Optical Flow Estimation using a Spatial Pyramid Network](https://mlsanity.com/paper/1611.00850/) (2016)
- [R-FCN: Object Detection via Region-based Fully Convolutional Networks](https://mlsanity.com/paper/1605.06409/) (2016)
- [BasicVSR++: Improving Video Super-Resolution with Enhanced Propagation and Alignment](https://mlsanity.com/paper/2104.13371/) (2021)
- [Super SloMo: High Quality Estimation of Multiple Intermediate Frames for Video Interpolation](https://mlsanity.com/paper/1712.00080/) (2017)
- [BasicVSR: The Search for Essential Components in Video Super-Resolution and Beyond](https://mlsanity.com/paper/2012.02181/) (2020)
- [Softmax Splatting for Video Frame Interpolation](https://mlsanity.com/paper/2003.05534/) (2020)
- [Depth-Aware Video Frame Interpolation](https://mlsanity.com/paper/1904.00830/) (2019)
- [Video Frame Interpolation via Adaptive Separable Convolution](https://mlsanity.com/paper/1708.01692/) (2017)
- [DeepStereo: Learning to Predict New Views from the World's Imagery](https://mlsanity.com/paper/1506.06825/) (2015)
- [Meta-SR: A Magnification-Arbitrary Network for Super-Resolution](https://mlsanity.com/paper/1903.00875/) (2019)
- [Frame-Recurrent Video Super-Resolution](https://mlsanity.com/paper/1801.04590/) (2018)
- [Real-Time Video Super-Resolution with Spatio-Temporal Networks and Motion Compensation](https://mlsanity.com/paper/1611.05250/) (2016)
- [Context-aware Synthesis for Video Frame Interpolation](https://mlsanity.com/paper/1803.10967/) (2018)
- [Detail-revealing Deep Video Super-resolution](https://mlsanity.com/paper/1704.02738/) (2017)
- [Recurrent Back-Projection Network for Video Super-Resolution](https://mlsanity.com/paper/1903.10128/) (2019)
- [Quadratic video interpolation](https://mlsanity.com/paper/1911.00627/) (2019)
- [XVFI: eXtreme Video Frame Interpolation](https://mlsanity.com/paper/2103.16206/) (2021)
- [Video Super-Resolution with Recurrent Structure-Detail Network](https://mlsanity.com/paper/2008.00455/) (2020)
- [Asymmetric Bilateral Motion Estimation for Video Frame Interpolation](https://mlsanity.com/paper/2108.06815/) (2021)
- [VideoINR: Learning Video Implicit Neural Representation for Continuous Space-Time Super-Resolution](https://mlsanity.com/paper/2206.04647/) (2022)
- [FLAVR: Flow-Agnostic Video Representations for Fast Frame Interpolation](https://mlsanity.com/paper/2012.08512/) (2020)
- [Models Matter, So Does Training: An Empirical Study of CNNs for Optical Flow Estimation](https://mlsanity.com/paper/1809.05571/) (2018)
- [Zooming Slow-Mo: Fast and Accurate One-Stage Space-Time Video Super-Resolution](https://mlsanity.com/paper/2002.11616/) (2020)

## Abstract (arXiv)

In this paper, we consider the task of space-time video super-resolution (ST-VSR), namely, expanding a given source video to a higher frame rate and resolution simultaneously. However, most existing schemes either consider a fixed intermediate time and scale in the training stage or only accept a preset number of input frames (e.g., two adjacent frames) that fails to exploit long-range temporal information. To address these problems, we propose a continuous ST-VSR (C-STVSR) method that can convert the given video to any frame rate and spatial resolution. To achieve time-arbitrary interpolation, we propose a forward warping guided frame synthesis module and an optical-flow-guided context consistency loss to better approximate extreme motion and preserve similar structures among input and prediction frames. In addition, we design a memory-friendly cascading depth-to-space module to realize continuous spatial upsampling. Meanwhile, with the sophisticated reorganization of optical flow, the proposed method is memory friendly, making it possible to propagate information from long-range neighboring frames and achieve better reconstruction quality. Extensive experiments show that the proposed algorithm has good flexibility and achieves better performance on various datasets compared with the state-of-the-art methods in both objective evaluations and subjective visual effects.