# Look Back and Forth: Video Super-Resolution with Explicit Temporal Difference Modeling

arXiv: https://arxiv.org/abs/2204.07114 · HTML: https://mlsanity.com/paper/2204.07114/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 46 references that are in the map.
- First posted on arXiv 14 Apr 2022.

## Builds on (most cited first, 30 of 35)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [Image Super-Resolution Using Very Deep Residual Channel Attention Networks](https://mlsanity.com/paper/1807.02758/) (2018)
- [Accurate Image Super-Resolution Using Very Deep Convolutional Networks](https://mlsanity.com/paper/1511.04587/) (2015)
- [Temporal Segment Networks: Towards Good Practices for Deep Action Recognition](https://mlsanity.com/paper/1608.00859/) (2016)
- [Video Enhancement with Task-Oriented Flow](https://mlsanity.com/paper/1711.09078/) (2017)
- [EDVR: Video Restoration with Enhanced Deformable Convolutional Networks](https://mlsanity.com/paper/1905.02716/) (2019)
- [Deep Laplacian Pyramid Networks for Fast and Accurate Super-Resolution](https://mlsanity.com/paper/1704.03915/) (2017)
- [Optical Flow Estimation using a Spatial Pyramid Network](https://mlsanity.com/paper/1611.00850/) (2016)
- [BasicVSR++: Improving Video Super-Resolution with Enhanced Propagation and Alignment](https://mlsanity.com/paper/2104.13371/) (2021)
- [Deeply-Recursive Convolutional Network for Image Super-Resolution](https://mlsanity.com/paper/1511.04491/) (2015)
- [BasicVSR: The Search for Essential Components in Video Super-Resolution and Beyond](https://mlsanity.com/paper/2012.02181/) (2020)
- [TDAN: Temporally Deformable Alignment Network for Video Super-Resolution](https://mlsanity.com/paper/1812.02898/) (2018)
- [Deep Back-Projection Networks For Super-Resolution](https://mlsanity.com/paper/1803.02735/) (2018)
- [TDN: Temporal Difference Networks for Efficient Action Recognition](https://mlsanity.com/paper/2012.10071/) (2020)
- [Frame-Recurrent Video Super-Resolution](https://mlsanity.com/paper/1801.04590/) (2018)
- [GLEAN: Generative Latent Bank for Large-Factor Image Super-Resolution](https://mlsanity.com/paper/2012.00739/) (2020)
- [Detail-revealing Deep Video Super-resolution](https://mlsanity.com/paper/1704.02738/) (2017)
- [Recurrent Back-Projection Network for Video Super-Resolution](https://mlsanity.com/paper/1903.10128/) (2019)
- [Video Super-Resolution Transformer](https://mlsanity.com/paper/2106.06847/) (2021)
- [Video Super-Resolution with Recurrent Structure-Detail Network](https://mlsanity.com/paper/2008.00455/) (2020)
- [MuCAN: Multi-Correspondence Aggregation Network for Video Super-Resolution](https://mlsanity.com/paper/2007.11803/) (2020)
- [Unsupervised Real-world Image Super Resolution via Domain-distance Aware Training](https://mlsanity.com/paper/2004.01178/) (2020)
- [Video Super-resolution with Temporal Group Attention](https://mlsanity.com/paper/2007.10595/) (2020)
- [Understanding Deformable Alignment in Video Super-Resolution](https://mlsanity.com/paper/2009.07265/) (2020)
- [Efficient Video Super-Resolution through Recurrent Latent Space Propagation](https://mlsanity.com/paper/1909.08080/) (2019)
- [Revisiting Temporal Modeling for Video Super-resolution](https://mlsanity.com/paper/2008.05765/) (2020)
- [Fast Spatio-Temporal Residual Network for Video Super-Resolution](https://mlsanity.com/paper/1904.02870/) (2019)
- [Deformable 3D Convolution for Video Super-Resolution](https://mlsanity.com/paper/2004.02803/) (2020)
- [Omniscient Video Super-Resolution](https://mlsanity.com/paper/2103.15683/) (2021)

## Built on by (most cited first, 20 of 20)

- [Video Super-Resolution Transformer with Masked Inter&Intra-Frame Attention](https://mlsanity.com/paper/2401.06312/) (2024)
- [Benchmark Dataset and Effective Inter-Frame Alignment for Real-World Video Super-Resolution](https://mlsanity.com/paper/2212.05342/) (2022)
- [Local-Global Temporal Difference Learning for Satellite Video Super-Resolution](https://mlsanity.com/paper/2304.04421/) (2023)
- [Mutual Information-Based Temporal Difference Learning for Human Pose Estimation in Video](https://mlsanity.com/paper/2303.08475/) (2023)
- [Structured Sparsity Learning for Efficient Video Super-Resolution](https://mlsanity.com/paper/2206.07687/) (2022)
- [LAVIB: A Large-scale Video Interpolation Benchmark](https://mlsanity.com/paper/2406.09754/) (2024)
- [TMP: Temporal Motion Propagation for Online Video Super-Resolution](https://mlsanity.com/paper/2312.09909/) (2023)
- [SupeRVol: Super-Resolution Shape and Reflectance Estimation in Inverse Volume Rendering](https://mlsanity.com/paper/2212.04968/) (2022)
- [Blind Video Super-Resolution based on Implicit Kernels](https://mlsanity.com/paper/2503.07856/) (2025)
- [SpatioTemporal Difference Network for Video Depth Super-Resolution](https://mlsanity.com/paper/2508.01259/) (2025)
- [Bilateral Event Mining and Complementary for Event Stream Super-Resolution](https://mlsanity.com/paper/2405.10037/) (2024)
- [Enhancing Space-time Video Super-resolution via Spatial-temporal Feature Interaction](https://mlsanity.com/paper/2207.08960/) (2022)
- [Collaborative Feedback Discriminative Propagation for Video Super-Resolution](https://mlsanity.com/paper/2404.04745/) (2024)
- [EgoVSR: Towards High-Quality Egocentric Video Super-Resolution](https://mlsanity.com/paper/2305.14708/) (2023)
- [Cascaded Temporal Updating Network for Efficient Video Super-Resolution](https://mlsanity.com/paper/2408.14244/) (2024)
- [Deep Learning Empowered Super-Resolution: A Comprehensive Survey and Future Prospects](https://mlsanity.com/paper/2509.22692/) (2025)
- [A Lightweight Recurrent Grouping Attention Network for Video Super-Resolution](https://mlsanity.com/paper/2309.13940/) (2023)
- [Small Clips, Big Gains: Learning Long-Range Refocused Temporal Information for Video Super-Resolution](https://mlsanity.com/paper/2505.02159/) (2025)
- [FCA2: Frame Compression-Aware Autoencoder for Modular and Fast Compressed Video Super-Resolution](https://mlsanity.com/paper/2506.11545/) (2025)
- [A Survey on Efficiency Optimization Techniques for DNN-based Video Analytics: Process Systems, Algorithms, and Applications](https://mlsanity.com/paper/2507.15628/) (2025)

## Abstract (arXiv)

Temporal modeling is crucial for video super-resolution. Most of the video super-resolution methods adopt the optical flow or deformable convolution for explicitly motion compensation. However, such temporal modeling techniques increase the model complexity and might fail in case of occlusion or complex motion, resulting in serious distortion and artifacts. In this paper, we propose to explore the role of explicit temporal difference modeling in both LR and HR space. Instead of directly feeding consecutive frames into a VSR model, we propose to compute the temporal difference between frames and divide those pixels into two subsets according to the level of difference. They are separately processed with two branches of different receptive fields in order to better extract complementary information. To further enhance the super-resolution result, not only spatial residual features are extracted, but the difference between consecutive frames in high-frequency domain is also computed. It allows the model to exploit intermediate SR results in both future and past to refine the current SR output. The difference at different time steps could be cached such that information from further distance in time could be propagated to the current frame for refinement. Experiments on several video super-resolution benchmark datasets demonstrate the effectiveness of the proposed method and its favorable performance against state-of-the-art methods.