# STAR: Spatial-Temporal Augmentation with Text-to-Video Models for Real-World Video Super-Resolution

arXiv: https://arxiv.org/abs/2501.02976 · HTML: https://mlsanity.com/paper/2501.02976/

## Key facts

- Cited by 56 papers in the map (data through 14 Oct 2026).
- Placed in Inverse problems and image restoration, one of 14 directions in Diffusion models.
- Builds on 66 of its 81 references that are in the map.
- First posted on arXiv 6 Jan 2025.

## Placed in

- Diffusion models / [Inverse problems and image restoration](https://mlsanity.com/diffusion/inverse-problems-and-image-restoration/)

## Builds on (most cited first, 30 of 66)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [Scalable Diffusion Models with Transformers](https://mlsanity.com/paper/2212.09748/) (2022)
- [Adding Conditional Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.05543/) (2023)
- [Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets](https://mlsanity.com/paper/2311.15127/) (2023)
- [CBAM: Convolutional Block Attention Module](https://mlsanity.com/paper/1807.06521/) (2018)
- [CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer](https://mlsanity.com/paper/2408.06072/) (2024)
- [SwinIR: Image Restoration Using Swin Transformer](https://mlsanity.com/paper/2108.10257/) (2021)
- [Imagen Video: High Definition Video Generation with Diffusion Models](https://mlsanity.com/paper/2210.02303/) (2022)
- [Make-A-Video: Text-to-Video Generation without Text-Video Data](https://mlsanity.com/paper/2209.14792/) (2022)
- [Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2304.08818/) (2023)
- [Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrieval](https://mlsanity.com/paper/2104.00650/) (2021)
- [Real-ESRGAN: Training Real-World Blind Super-Resolution with Pure Synthetic Data](https://mlsanity.com/paper/2107.10833/) (2021)
- [Movie Gen: A Cast of Media Foundation Models](https://mlsanity.com/paper/2410.13720/) (2024)
- [Video Enhancement with Task-Oriented Flow](https://mlsanity.com/paper/1711.09078/) (2017)
- [The Perception-Distortion Tradeoff](https://mlsanity.com/paper/1711.06077/) (2017)
- [Scaling Laws for Autoregressive Generative Modeling](https://mlsanity.com/paper/2010.14701/) (2020)
- [InternVid: A Large-scale Video-Text Dataset for Multimodal Understanding and Generation](https://mlsanity.com/paper/2307.06942/) (2023)
- [EDVR: Video Restoration with Enhanced Deformable Convolutional Networks](https://mlsanity.com/paper/1905.02716/) (2019)
- [Exploiting Diffusion Prior for Real-World Image Super-Resolution](https://mlsanity.com/paper/2305.07015/) (2023)
- [Panda-70M: Captioning 70M Videos with Multiple Cross-Modality Teachers](https://mlsanity.com/paper/2402.19479/) (2024)
- [ResShift: Efficient Diffusion Model for Image Super-resolution by Residual Shifting](https://mlsanity.com/paper/2307.12348/) (2023)
- [I2VGen-XL: High-Quality Image-to-Video Synthesis via Cascaded Diffusion Models](https://mlsanity.com/paper/2311.04145/) (2023)
- [BasicVSR++: Improving Video Super-Resolution with Enhanced Propagation and Alignment](https://mlsanity.com/paper/2104.13371/) (2021)
- [OpenVid-1M: A Large-Scale High-Quality Dataset for Text-to-video Generation](https://mlsanity.com/paper/2407.02371/) (2024)
- [EvalCrafter: Benchmarking and Evaluating Large Video Generation Models](https://mlsanity.com/paper/2310.11440/) (2023)
- [VideoFusion: Decomposed Diffusion Models for High-Quality Video Generation](https://mlsanity.com/paper/2303.08320/) (2023)

## Built on by (most cited first, 30 of 56)

- [SeedVR2: One-Step Video Restoration via Diffusion Adversarial Post-Training](https://mlsanity.com/paper/2506.05301/) (2025)
- [FlashVSR: Towards Real-Time Diffusion-Based Streaming Video Super-Resolution](https://mlsanity.com/paper/2510.12747/) (2025)
- [DOVE: Efficient One-Step Diffusion Model for Real-World Video Super-Resolution](https://mlsanity.com/paper/2505.16239/) (2025)
- [InteractiveOmni: A Unified Omni-modal Model for Audio-Visual Multi-turn Dialogue](https://mlsanity.com/paper/2510.13747/) (2025)
- [The Eleventh NTIRE 2026 Efficient Super-Resolution Challenge Report](https://mlsanity.com/paper/2604.03198/) (2026)
- [UltraHR-100K: Enhancing UHR Image Synthesis with A Large-Scale High-Quality Dataset](https://mlsanity.com/paper/2510.20661/) (2025)
- [One-Step Diffusion for Detail-Rich and Temporally Consistent Video Super-Resolution](https://mlsanity.com/paper/2506.15591/) (2025)
- [LUVE : Latent-Cascaded Ultra-High-Resolution Video Generation with Dual Frequency Experts](https://mlsanity.com/paper/2602.11564/) (2026)
- [Turbo2K: Towards Ultra-Efficient and High-Quality 2K Video Synthesis](https://mlsanity.com/paper/2504.14470/) (2025)
- [Vivid-VR: Distilling Concepts from Text-to-Video Diffusion Transformer for Photorealistic Video Restoration](https://mlsanity.com/paper/2508.14483/) (2025)
- [InfVSR: Toward Consistency-Driven Streaming Generative Video Super-Resolution](https://mlsanity.com/paper/2510.00948/) (2025)
- [Stream-DiffVSR: Low-Latency Streamable Video Super-Resolution via Auto-Regressive Diffusion](https://mlsanity.com/paper/2512.23709/) (2025)
- [LiftVSR: Lifting Image Diffusion to Video Super-Resolution via Hybrid Temporal Modeling with Only 4×RTX 4090s](https://mlsanity.com/paper/2506.08529/) (2025)
- [SimpleGVR: A Simple Baseline for Latent-Cascaded Video Super-Resolution](https://mlsanity.com/paper/2506.19838/) (2025)
- [Improved Adversarial Diffusion Compression for Real-World Video Super-Resolution](https://mlsanity.com/paper/2603.00458/) (2026)
- [SparkVSR: Interactive Video Super-Resolution via Sparse Keyframe Propagation](https://mlsanity.com/paper/2603.16864/) (2026)
- [Many-for-Many: Unify the Training of Multiple Video and Image Generation and Manipulation Tasks](https://mlsanity.com/paper/2506.01758/) (2025)
- [DUO-VSR: Dual-Stream Distillation for One-Step Video Super-Resolution](https://mlsanity.com/paper/2603.22271/) (2026)
- [OS-DiffVSR: Towards One-step Latent Diffusion Model for High-detailed Real-world Video Super-Resolution](https://mlsanity.com/paper/2509.16507/) (2025)
- [Asymmetric VAE for One-Step Video Super-Resolution Acceleration](https://mlsanity.com/paper/2509.24142/) (2025)
- [TurboVSR: Fantastic Video Upscalers and Where to Find Them](https://mlsanity.com/paper/2506.23618/) (2025)
- [Towards Redundancy Reduction in Diffusion Models for Efficient Video Super-Resolution](https://mlsanity.com/paper/2509.23980/) (2025)
- [Rethinking Diffusion Model-Based Video Super-Resolution: Leveraging Dense Guidance from Aligned Features](https://mlsanity.com/paper/2511.16928/) (2025)
- [CreativeVR: Diffusion-Prior-Guided Approach for Structure and Motion Restoration in Generative and Real Videos](https://mlsanity.com/paper/2512.12060/) (2025)
- [PixelWizard: Towards Efficient High-Fidelity Video Generation at Ultra-Large Spatial Resolution](https://mlsanity.com/paper/2605.25801/) (2026)
- [SwiftVR: Real-Time One-Step Generative Video Restoration](https://mlsanity.com/paper/2606.09516/) (2026)
- [Accelerating Diffusion-based Super-Resolution with Dynamic Time-Spatial Sampling](https://mlsanity.com/paper/2505.12048/) (2025)
- [UniMMVSR: A Unified Multi-Modal Framework for Cascaded Video Super-Resolution](https://mlsanity.com/paper/2510.08143/) (2025)
- [STCDiT: Spatio-Temporally Consistent Diffusion Transformer for High-Quality Video Super-Resolution](https://mlsanity.com/paper/2511.18786/) (2025)
- [Taming Real-World Space-Time Video Super-Resolution with One-Step Diffusion](https://mlsanity.com/paper/2601.20308/) (2026)

## Abstract (arXiv)

Image diffusion models have been adapted for real-world video super-resolution to tackle over-smoothing issues in GAN-based methods. However, these models struggle to maintain temporal consistency, as they are trained on static images, limiting their ability to capture temporal dynamics effectively. Integrating text-to-video (T2V) models into video super-resolution for improved temporal modeling is straightforward. However, two key challenges remain: artifacts introduced by complex degradations in real-world scenarios, and compromised fidelity due to the strong generative capacity of powerful T2V models (textite.g., CogVideoX-5B). To enhance the spatio-temporal quality of restored videos, we introduce~name (Spatial-Temporal Augmentation with T2V models for Real-world video super-resolution), a novel approach that leverages T2V models for real-world video super-resolution, achieving realistic spatial details and robust temporal consistency. Specifically, we introduce a Local Information Enhancement Module (LIEM) before the global attention block to enrich local details and mitigate degradation artifacts. Moreover, we propose a Dynamic Frequency (DF) Loss to reinforce fidelity, guiding the model to focus on different frequency components across diffusion steps. Extensive experiments demonstrate~name~outperforms state-of-the-art methods on both synthetic and real-world datasets.