# Understanding Deformable Alignment in Video Super-Resolution

arXiv: https://arxiv.org/abs/2009.07265 · HTML: https://mlsanity.com/paper/2009.07265/

## Key facts

- Cited by 78 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 15 Sep 2020.

## Built on by (most cited first, 30 of 78)

- [BasicVSR++: Improving Video Super-Resolution with Enhanced Propagation and Alignment](https://mlsanity.com/paper/2104.13371/) (2021)
- [VRT: A Video Restoration Transformer](https://mlsanity.com/paper/2201.12288/) (2022)
- [Recurrent Video Restoration Transformer with Guided Deformable Attention](https://mlsanity.com/paper/2206.02146/) (2022)
- [Neural Video Compression with Diverse Contexts](https://mlsanity.com/paper/2302.14402/) (2023)
- [Investigating Tradeoffs in Real-World Video Super-Resolution](https://mlsanity.com/paper/2111.12704/) (2021)
- [ALIKED: A Lighter Keypoint and Descriptor Extraction Network via Deformable Transformation](https://mlsanity.com/paper/2304.03608/) (2023)
- [Towards An End-to-End Framework for Flow-Guided Video Inpainting](https://mlsanity.com/paper/2204.02663/) (2022)
- [Rethinking Alignment in Video Super-Resolution Transformers](https://mlsanity.com/paper/2207.08494/) (2022)
- [Robust Reference-based Super-Resolution via C2-Matching](https://mlsanity.com/paper/2106.01863/) (2021)
- [A Simple Baseline for Video Restoration with Grouped Spatial-temporal Shift](https://mlsanity.com/paper/2206.10810/) (2022)
- [Attentions Help CNNs See Better: Attention-based Hybrid Image Quality Assessment Network](https://mlsanity.com/paper/2204.10485/) (2022)
- [ADNet: Attention-guided Deformable Convolutional Network for High Dynamic Range Imaging](https://mlsanity.com/paper/2105.10697/) (2021)
- [High-Fidelity and Freely Controllable Talking Head Video Generation](https://mlsanity.com/paper/2304.10168/) (2023)
- [Dual-Camera Super-Resolution with Aligned Attention Modules](https://mlsanity.com/paper/2109.01349/) (2021)
- [DiffIR2VR-Zero: Zero-Shot Video Restoration with Diffusion-based Image Restoration Models](https://mlsanity.com/paper/2407.01519/) (2024)
- [FMA-Net: Flow-Guided Dynamic Filtering and Iterative Feature Refinement with Multi-Attention for Joint Video Super-Resolution and Deblurring](https://mlsanity.com/paper/2401.03707/) (2024)
- [NTIRE 2021 Challenge on Quality Enhancement of Compressed Video: Methods and Results](https://mlsanity.com/paper/2104.10781/) (2021)
- [Self-Supervised Learning for Real-World Super-Resolution from Dual Zoomed Observations](https://mlsanity.com/paper/2203.01325/) (2022)
- [Look Back and Forth: Video Super-Resolution with Explicit Temporal Difference Modeling](https://mlsanity.com/paper/2204.07114/) (2022)
- [Benchmark Dataset and Effective Inter-Frame Alignment for Real-World Video Super-Resolution](https://mlsanity.com/paper/2212.05342/) (2022)
- [FDAN: Flow-guided Deformable Alignment Network for Video Super-Resolution](https://mlsanity.com/paper/2105.05640/) (2021)
- [On the Generalization of BasicVSR++ to Video Deblurring and Denoising](https://mlsanity.com/paper/2204.05308/) (2022)
- [Temporal Consistency Learning of inter-frames for Video Super-Resolution](https://mlsanity.com/paper/2211.01639/) (2022)
- [Recursive Fusion and Deformable Spatiotemporal Attention for Video Compression Artifact Reduction](https://mlsanity.com/paper/2108.02110/) (2021)
- [Multi-task Image Restoration Guided By Robust DINO Features](https://mlsanity.com/paper/2312.01677/) (2023)
- [Spatial-Temporal Space Hand-in-Hand: Spatial-Temporal Video Super-Resolution via Cycle-Projected Mutual Learning](https://mlsanity.com/paper/2205.05264/) (2022)
- [Video Frame Interpolation with Stereo Event and Intensity Camera](https://mlsanity.com/paper/2307.08228/) (2023)
- [Reference-based Image and Video Super-Resolution via C2-Matching](https://mlsanity.com/paper/2212.09581/) (2022)
- [Scale-Adaptive Feature Aggregation for Efficient Space-Time Video Super-Resolution](https://mlsanity.com/paper/2310.17294/) (2023)
- [TMP: Temporal Motion Propagation for Online Video Super-Resolution](https://mlsanity.com/paper/2312.09909/) (2023)

## Abstract (arXiv)

Deformable convolution, originally proposed for the adaptation to geometric variations of objects, has recently shown compelling performance in aligning multiple frames and is increasingly adopted for video super-resolution. Despite its remarkable performance, its underlying mechanism for alignment remains unclear. In this study, we carefully investigate the relation between deformable alignment and the classic flow-based alignment. We show that deformable convolution can be decomposed into a combination of spatial warping and convolution. This decomposition reveals the commonality of deformable alignment and flow-based alignment in formulation, but with a key difference in their offset diversity. We further demonstrate through experiments that the increased diversity in deformable alignment yields better-aligned features, and hence significantly improves the quality of video super-resolution output. Based on our observations, we propose an offset-fidelity loss that guides the offset learning with optical flow. Experiments show that our loss successfully avoids the overflow of offsets and alleviates the instability problem of deformable alignment. Aside from the contributions to deformable alignment, our formulation inspires a more flexible approach to introduce offset diversity to flow-based alignment, improving its performance.