# Learning Spatiotemporal Frequency-Transformer for Compressed Video Super-Resolution

arXiv: https://arxiv.org/abs/2208.03012 · HTML: https://mlsanity.com/paper/2208.03012/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 39 references that are in the map.
- First posted on arXiv 5 Aug 2022.

## Builds on (most cited first, 27 of 27)

- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Video Enhancement with Task-Oriented Flow](https://mlsanity.com/paper/1711.09078/) (2017)
- [EDVR: Video Restoration with Enhanced Deformable Convolutional Networks](https://mlsanity.com/paper/1905.02716/) (2019)
- [Deep Laplacian Pyramid Networks for Fast and Accurate Super-Resolution](https://mlsanity.com/paper/1704.03915/) (2017)
- [Learning Texture Transformer Network for Image Super-Resolution](https://mlsanity.com/paper/2006.04139/) (2020)
- [BasicVSR: The Search for Essential Components in Video Super-Resolution and Beyond](https://mlsanity.com/paper/2012.02181/) (2020)
- [TDAN: Temporally Deformable Alignment Network for Video Super-Resolution](https://mlsanity.com/paper/1812.02898/) (2018)
- [Learning in the Frequency Domain](https://mlsanity.com/paper/2002.12416/) (2020)
- [FcaNet: Frequency Channel Attention Networks](https://mlsanity.com/paper/2012.11879/) (2020)
- [Frame-Recurrent Video Super-Resolution](https://mlsanity.com/paper/1801.04590/) (2018)
- [Detail-revealing Deep Video Super-resolution](https://mlsanity.com/paper/1704.02738/) (2017)
- [Recurrent Back-Projection Network for Video Super-Resolution](https://mlsanity.com/paper/1903.10128/) (2019)
- [Video Super-Resolution Transformer](https://mlsanity.com/paper/2106.06847/) (2021)
- [Video Super-Resolution with Recurrent Structure-Detail Network](https://mlsanity.com/paper/2008.00455/) (2020)
- [Frequency Separation for Real-World Super-Resolution](https://mlsanity.com/paper/1911.07850/) (2019)
- [MuCAN: Multi-Correspondence Aggregation Network for Video Super-Resolution](https://mlsanity.com/paper/2007.11803/) (2020)
- [Learning Trajectory-Aware Transformer for Video Super-Resolution](https://mlsanity.com/paper/2204.04216/) (2022)
- [Quantization Guided JPEG Artifact Correction](https://mlsanity.com/paper/2004.09320/) (2020)
- [Deep Residual Learning in the JPEG Transform Domain](https://mlsanity.com/paper/1812.11690/) (2018)
- [Fast Spatio-Temporal Residual Network for Video Super-Resolution](https://mlsanity.com/paper/1904.02870/) (2019)
- [Omniscient Video Super-Resolution](https://mlsanity.com/paper/2103.15683/) (2021)
- [COMISR: Compression-Informed Video Super-Resolution](https://mlsanity.com/paper/2105.01237/) (2021)
- [D3: Deep Dual-Domain Based Fast Restoration of JPEG-Compressed Images](https://mlsanity.com/paper/1601.04149/) (2016)
- [Non-Local ConvLSTM for Video Compression Artifact Reduction](https://mlsanity.com/paper/1910.12286/) (2019)
- [Improving Visual Quality of Image Synthesis by A Token-based Generator with Transformers](https://mlsanity.com/paper/2111.03481/) (2021)
- [Learning Omni-frequency Region-adaptive Representations for Real Image Super-Resolution](https://mlsanity.com/paper/2012.06131/) (2020)
- [3DSRnet: Video Super-resolution using 3D Convolutional Neural Networks](https://mlsanity.com/paper/1812.09079/) (2018)

## Built on by (most cited first, 25 of 25)

- [MM-Diffusion: Learning Multi-Modal Diffusion Models for Joint Audio and Video Generation](https://mlsanity.com/paper/2212.09478/) (2022)
- [Zero-Reference Low-Light Enhancement via Physical Quadruple Priors](https://mlsanity.com/paper/2403.12933/) (2024)
- [MovieFactory: Automatic Movie Creation from Text using Large Generative Models for Language and Images](https://mlsanity.com/paper/2306.07257/) (2023)
- [Video Super-Resolution Transformer with Masked Inter&Intra-Frame Attention](https://mlsanity.com/paper/2401.06312/) (2024)
- [FMA-Net: Flow-Guided Dynamic Filtering and Iterative Feature Refinement with Multi-Attention for Joint Video Super-Resolution and Deblurring](https://mlsanity.com/paper/2401.03707/) (2024)
- [Fine-Grained Image Style Transfer with Visual Transformers](https://mlsanity.com/paper/2210.05176/) (2022)
- [OmniCT: Towards a Unified Slice-Volume LVLM for Comprehensive CT Analysis](https://mlsanity.com/paper/2602.16110/) (2026)
- [Online Streaming Video Super-Resolution with Convolutional Look-Up Table](https://mlsanity.com/paper/2303.00334/) (2023)
- [Learning Profitable NFT Image Diffusions via Multiple Visual-Policy Guided Reinforcement Learning](https://mlsanity.com/paper/2306.11731/) (2023)
- [Exploiting Optical Flow Guidance for Transformer-Based Video Inpainting](https://mlsanity.com/paper/2301.10048/) (2023)
- [A Hybrid Transformer-Mamba Network for Single Image Deraining](https://mlsanity.com/paper/2409.00410/) (2024)
- [Learning Structure-Guided Diffusion Model for 2D Human Pose Estimation](https://mlsanity.com/paper/2306.17074/) (2023)
- [Learning Trimodal Relation for Audio-Visual Question Answering with Missing Modality](https://mlsanity.com/paper/2407.16171/) (2024)
- [Learning Data-Driven Vector-Quantized Degradation Model for Animation Video Super-Resolution](https://mlsanity.com/paper/2303.09826/) (2023)
- [Continuous Space-Time Video Super-Resolution with 3D Fourier Fields](https://mlsanity.com/paper/2509.26325/) (2025)
- [Reversing the Damage: A QP-Aware Transformer-Diffusion Approach for 8K Video Restoration under Codec Compression](https://mlsanity.com/paper/2412.08912/) (2024)
- [FCVSR: A Frequency-aware Method for Compressed Video Super-Resolution](https://mlsanity.com/paper/2502.06431/) (2025)
- [Spatial Degradation-Aware and Temporal Consistent Diffusion Model for Compressed Video Super-Resolution](https://mlsanity.com/paper/2502.07381/) (2025)
- [Regulating Anatomy-Aware Rewards via Trajectory-Integral Feedback for Volumetric Computed Tomography Analysis](https://mlsanity.com/paper/2605.20277/) (2026)
- [Learning Spatiotemporal Frequency-Transformer for Low-Quality Video Super-Resolution](https://mlsanity.com/paper/2212.14046/) (2022)
- [Hierarchical Frequency-based Upsampling and Refining for Compressed Video Quality Enhancement](https://mlsanity.com/paper/2403.11556/) (2024)
- [Small Clips, Big Gains: Learning Long-Range Refocused Temporal Information for Video Super-Resolution](https://mlsanity.com/paper/2505.02159/) (2025)
- [FCA2: Frame Compression-Aware Autoencoder for Modular and Fast Compressed Video Super-Resolution](https://mlsanity.com/paper/2506.11545/) (2025)
- [Exploring Real-Time Super-Resolution: Benchmarking and Fine-Tuning for Streaming Content](https://mlsanity.com/paper/2602.11339/) (2026)
- [Diffusion Models for Joint Audio-Video Generation](https://mlsanity.com/paper/2603.16093/) (2026)

## Abstract (arXiv)

Compressed video super-resolution (VSR) aims to restore high-resolution frames from compressed low-resolution counterparts. Most recent VSR approaches often enhance an input frame by borrowing relevant textures from neighboring video frames. Although some progress has been made, there are grand challenges to effectively extract and transfer high-quality textures from compressed videos where most frames are usually highly degraded. In this paper, we propose a novel Frequency-Transformer for compressed video super-resolution (FTVSR) that conducts self-attention over a joint space-time-frequency domain. First, we divide a video frame into patches, and transform each patch into DCT spectral maps in which each channel represents a frequency band. Such a design enables a fine-grained level self-attention on each frequency band, so that real visual texture can be distinguished from artifacts, and further utilized for video frame restoration. Second, we study different self-attention schemes, and discover that a divided attention which conducts a joint space-frequency attention before applying temporal attention on each frequency band, leads to the best video enhancement quality. Experimental results on two widely-used video super-resolution benchmarks show that FTVSR outperforms state-of-the-art approaches on both uncompressed and compressed videos with clear visual margins. Code is available at https://github.com/researchmm/FTVSR.