# Across Scales & Across Dimensions: Temporal Super-Resolution using Deep Internal Learning

arXiv: https://arxiv.org/abs/2003.08872 · HTML: https://mlsanity.com/paper/2003.08872/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 19 Mar 2020.

## Built on by (most cited first, 7 of 7)

- [Towards Interpretable Video Super-Resolution via Alternating Optimization](https://mlsanity.com/paper/2207.10765/) (2022)
- [Deep Neural Network-based Enhancement for Image and Video Streaming Systems: A Survey and Future Directions](https://mlsanity.com/paper/2106.03727/) (2021)
- [Zooming SlowMo: An Efficient One-Stage Framework for Space-Time Video Super-Resolution](https://mlsanity.com/paper/2104.07473/) (2021)
- [Combining Internal and External Constraints for Unrolling Shutter in Videos](https://mlsanity.com/paper/2207.11725/) (2022)
- [Learning Spatio-Temporal Downsampling for Effective Video Upscaling](https://mlsanity.com/paper/2203.08140/) (2022)
- [Connect the Dots: In Situ 4D Seismic Monitoring of CO2 Storage with Spatio-temporal CNNs](https://mlsanity.com/paper/2105.11622/) (2021)
- [Self-Supervised Adaptation for Video Super-Resolution](https://mlsanity.com/paper/2103.10081/) (2021)

## Abstract (arXiv)

When a very fast dynamic event is recorded with a low-framerate camera, the resulting video suffers from severe motion blur (due to exposure time) and motion aliasing (due to low sampling rate in time). True Temporal Super-Resolution (TSR) is more than just Temporal-Interpolation (increasing framerate). It can also recover new high temporal frequencies beyond the temporal Nyquist limit of the input video, thus resolving both motion-blur and motion-aliasing effects that temporal frame interpolation (as sophisticated as it maybe) cannot undo. In this paper we propose a "Deep Internal Learning" approach for true TSR. We train a video-specific CNN on examples extracted directly from the low-framerate input video. Our method exploits the strong recurrence of small space-time patches inside a single video sequence, both within and across different spatio-temporal scales of the video. We further observe (for the first time) that small space-time patches recur also across-dimensions of the video sequence - i.e., by swapping the spatial and temporal dimensions. In particular, the higher spatial resolution of video frames provides strong examples as to how to increase the temporal resolution of that video. Such internal video-specific examples give rise to strong self-supervision, requiring no data but the input video itself. This results in Zero-Shot Temporal-SR of complex videos, which removes both motion blur and motion aliasing, outperforming previous supervised methods trained on external video datasets.