# Can SAM Boost Video Super-Resolution?

arXiv: https://arxiv.org/abs/2305.06524 · HTML: https://mlsanity.com/paper/2305.06524/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Builds on 37 of its 70 references that are in the map.
- First posted on arXiv 11 May 2023.

## Builds on (most cited first, 30 of 37)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Learning to Prompt for Vision-Language Models](https://mlsanity.com/paper/2109.01134/) (2021)
- [Visual Prompt Tuning](https://mlsanity.com/paper/2203.12119/) (2022)
- [Conditional Prompt Learning for Vision-Language Models](https://mlsanity.com/paper/2203.05557/) (2022)
- [Image Super-Resolution Using Very Deep Residual Channel Attention Networks](https://mlsanity.com/paper/1807.02758/) (2018)
- [Deep Image Prior](https://mlsanity.com/paper/1711.10925/) (2017)
- [CLIP-Adapter: Better Vision-Language Models with Feature Adapters](https://mlsanity.com/paper/2110.04544/) (2021)
- [PWC-Net: CNNs for Optical Flow Using Pyramid, Warping, and Cost Volume](https://mlsanity.com/paper/1709.02371/) (2017)
- [How Can We Know What Language Models Know?](https://mlsanity.com/paper/1911.12543/) (2019)
- [SpanBERT: Improving Pre-training by Representing and Predicting Spans](https://mlsanity.com/paper/1907.10529/) (2019)
- [Video Enhancement with Task-Oriented Flow](https://mlsanity.com/paper/1711.09078/) (2017)
- [EDVR: Video Restoration with Enhanced Deformable Convolutional Networks](https://mlsanity.com/paper/1905.02716/) (2019)
- [Optical Flow Estimation using a Spatial Pyramid Network](https://mlsanity.com/paper/1611.00850/) (2016)
- [Tip-Adapter: Training-free Adaption of CLIP for Few-shot Classification](https://mlsanity.com/paper/2207.09519/) (2022)
- [SinGAN: Learning a Generative Model from a Single Natural Image](https://mlsanity.com/paper/1905.01164/) (2019)
- [BasicVSR++: Improving Video Super-Resolution with Enhanced Propagation and Alignment](https://mlsanity.com/paper/2104.13371/) (2021)
- [BasicVSR: The Search for Essential Components in Video Super-Resolution and Beyond](https://mlsanity.com/paper/2012.02181/) (2020)
- [TDAN: Temporally Deformable Alignment Network for Video Super-Resolution](https://mlsanity.com/paper/1812.02898/) (2018)
- [Trainable Nonlinear Reaction Diffusion: A Flexible Framework for Fast and Effective Image Restoration](https://mlsanity.com/paper/1508.02848/) (2015)
- [Frame-Recurrent Video Super-Resolution](https://mlsanity.com/paper/1801.04590/) (2018)
- [Real-Time Video Super-Resolution with Spatio-Temporal Networks and Motion Compensation](https://mlsanity.com/paper/1611.05250/) (2016)
- [Detail-revealing Deep Video Super-resolution](https://mlsanity.com/paper/1704.02738/) (2017)
- [Recurrent Back-Projection Network for Video Super-Resolution](https://mlsanity.com/paper/1903.10128/) (2019)
- [Task Residual for Tuning Vision-Language Models](https://mlsanity.com/paper/2211.10277/) (2022)
- [Video Super-Resolution Transformer](https://mlsanity.com/paper/2106.06847/) (2021)
- [Video Super-Resolution with Recurrent Structure-Detail Network](https://mlsanity.com/paper/2008.00455/) (2020)

## Built on by (most cited first, 21 of 21)

- [LangSplat: 3D Language Gaussian Splatting](https://mlsanity.com/paper/2312.16084/) (2023)
- [A Comprehensive Survey on Segment Anything Model for Vision and Beyond](https://mlsanity.com/paper/2305.08196/) (2023)
- [A Survey on Segment Anything Model (SAM): Vision Foundation Model Meets Prompt Engineering](https://mlsanity.com/paper/2306.06211/) (2023)
- [RobustSAM: Segment Anything Robustly on Degraded Images](https://mlsanity.com/paper/2406.09627/) (2024)
- [Every SAM Drop Counts: Embracing Semantic Priors for Multi-Modality Image Fusion and Beyond](https://mlsanity.com/paper/2503.01210/) (2025)
- [Transfer CLIP for Generalizable Image Denoising](https://mlsanity.com/paper/2403.15132/) (2024)
- [SlimSAM: 0.1% Data Makes Segment Anything Slim](https://mlsanity.com/paper/2312.05284/) (2023)
- [PA-SAM: Prompt Adapter SAM for High-Quality Image Segmentation](https://mlsanity.com/paper/2401.13051/) (2024)
- [On Efficient Variants of Segment Anything Model: A Survey](https://mlsanity.com/paper/2410.04960/) (2024)
- [ROS-SAM: High-Quality Interactive Segmentation for Remote Sensing Moving Object](https://mlsanity.com/paper/2503.12006/) (2025)
- [SAM-DiffSR: Structure-Modulated Diffusion Model for Image Super-Resolution](https://mlsanity.com/paper/2402.17133/) (2024)
- [A Dive into SAM Prior in Image Restoration](https://mlsanity.com/paper/2305.13620/) (2023)
- [TMP: Temporal Motion Propagation for Online Video Super-Resolution](https://mlsanity.com/paper/2312.09909/) (2023)
- [Segment Anything for Videos: A Systematic Survey](https://mlsanity.com/paper/2408.08315/) (2024)
- [Industrial Segment Anything -- a Case Study in Aircraft Manufacturing, Intralogistics, Maintenance, Repair, and Overhaul](https://mlsanity.com/paper/2307.12674/) (2023)
- [Evaluating SAM2 for Video Semantic Segmentation](https://mlsanity.com/paper/2512.01774/) (2025)
- [Semantic-Guided Diffusion Model for Single-Step Image Super-Resolution](https://mlsanity.com/paper/2505.07071/) (2025)
- [SAM-IQA: Can Segment Anything Boost Image Quality Assessment?](https://mlsanity.com/paper/2307.04455/) (2023)
- [TriSAM: Tri-Plane SAM for zero-shot cortical blood vessel segmentation in VEM images](https://mlsanity.com/paper/2401.13961/) (2024)
- [Restoration-Oriented Video Frame Interpolation with Region-Distinguishable Priors from SAM](https://mlsanity.com/paper/2312.15868/) (2023)
- [Mitigating Hallucinations on Object Attributes using Multiview Images and Negative Instructions](https://mlsanity.com/paper/2501.10011/) (2025)

## Abstract (arXiv)

The primary challenge in video super-resolution (VSR) is to handle large motions in the input frames, which makes it difficult to accurately aggregate information from multiple frames. Existing works either adopt deformable convolutions or estimate optical flow as a prior to establish correspondences between frames for the effective alignment and fusion. However, they fail to take into account the valuable semantic information that can greatly enhance it; and flow-based methods heavily rely on the accuracy of a flow estimate model, which may not provide precise flows given two low-resolution frames. In this paper, we investigate a more robust and semantic-aware prior for enhanced VSR by utilizing the Segment Anything Model (SAM), a powerful foundational model that is less susceptible to image degradation. To use the SAM-based prior, we propose a simple yet effective module -- SAM-guidEd refinEment Module (SEEM), which can enhance both alignment and fusion procedures by the utilization of semantic information. This light-weight plug-in module is specifically designed to not only leverage the attention mechanism for the generation of semantic-aware feature but also be easily and seamlessly integrated into existing methods. Concretely, we apply our SEEM to two representative methods, EDVR and BasicVSR, resulting in consistently improved performance with minimal implementation effort, on three widely used VSR datasets: Vimeo-90K, REDS and Vid4. More importantly, we found that the proposed SEEM can advance the existing methods in an efficient tuning manner, providing increased flexibility in adjusting the balance between performance and the number of training parameters. Code will be open-source soon.