# Exploring Spatial-Temporal Multi-Frequency Analysis for High-Fidelity and Temporal-Consistency Video Prediction

arXiv: https://arxiv.org/abs/2002.09905 · HTML: https://mlsanity.com/paper/2002.09905/

## Key facts

- Cited by 52 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 Feb 2020.

## Built on by (most cited first, 30 of 52)

- [MCVD: Masked Conditional Video Diffusion for Prediction, Generation, and Interpolation](https://mlsanity.com/paper/2205.09853/) (2022)
- [SimVP: Simpler yet Better Video Prediction](https://mlsanity.com/paper/2206.05099/) (2022)
- [Temporal Attention Unit: Towards Efficient Spatiotemporal Predictive Learning](https://mlsanity.com/paper/2206.12126/) (2022)
- [Video Prediction Recalling Long-term Motion Context via Memory Alignment Learning](https://mlsanity.com/paper/2104.00924/) (2021)
- [Iso-Dream: Isolating and Leveraging Noncontrollable Visual Dynamics in World Models](https://mlsanity.com/paper/2205.13817/) (2022)
- [Understanding Object Dynamics for Interactive Image-to-Video Synthesis](https://mlsanity.com/paper/2106.11303/) (2021)
- [VMRNN: Integrating Vision Mamba and LSTM for Efficient and Accurate Spatiotemporal Forecasting](https://mlsanity.com/paper/2403.16536/) (2024)
- [SwinLSTM:Improving Spatiotemporal Prediction Accuracy using Swin Transformer and LSTM](https://mlsanity.com/paper/2308.09891/) (2023)
- [STRPM: A Spatiotemporal Residual Predictive Model for High-Resolution Video Prediction](https://mlsanity.com/paper/2203.16084/) (2022)
- [SimVPv2: Towards Simple yet Powerful Spatiotemporal Predictive Learning](https://mlsanity.com/paper/2211.12509/) (2022)
- [Learning Semantic-Aware Dynamics for Video Prediction](https://mlsanity.com/paper/2104.09762/) (2021)
- [MMVP: Motion-Matrix-based Video Prediction](https://mlsanity.com/paper/2308.16154/) (2023)
- [VPTR: Efficient Transformers for Video Prediction](https://mlsanity.com/paper/2203.15836/) (2022)
- [Advancing Learned Video Compression with In-loop Frame Prediction](https://mlsanity.com/paper/2211.07004/) (2022)
- [Accurate Grid Keypoint Learning for Efficient Video Prediction](https://mlsanity.com/paper/2107.13170/) (2021)
- [Video Prediction by Efficient Transformers](https://mlsanity.com/paper/2212.06026/) (2022)
- [STDiff: Spatio-temporal Diffusion for Continuous Stochastic Video Prediction](https://mlsanity.com/paper/2312.06486/) (2023)
- [Future Frame Prediction for Robot-assisted Surgery](https://mlsanity.com/paper/2103.10308/) (2021)
- [PCPNet: An Efficient and Semantic-Enhanced Transformer Network for Point Cloud Prediction](https://mlsanity.com/paper/2304.07773/) (2023)
- [A Hierarchical Variational Neural Uncertainty Model for Stochastic Video Prediction](https://mlsanity.com/paper/2110.03446/) (2021)
- [SNeRV: Spectra-preserving Neural Representation for Video](https://mlsanity.com/paper/2501.01681/) (2025)
- [Physics-informed generative neural network: an application to troposphere temperature prediction](https://mlsanity.com/paper/2107.06991/) (2021)
- [Two-stage Rule-induction Visual Reasoning on RPMs with an Application to Video Prediction](https://mlsanity.com/paper/2111.12301/) (2021)
- [A unified model for continuous conditional video prediction](https://mlsanity.com/paper/2210.05810/) (2022)
- [S-HR-VQVAE: Sequential Hierarchical Residual Learning Vector Quantized Variational Autoencoder for Video Prediction](https://mlsanity.com/paper/2307.06701/) (2023)
- [ModeRNN: Harnessing Spatiotemporal Mode Collapse in Unsupervised Predictive Learning](https://mlsanity.com/paper/2110.03882/) (2021)
- [Model-Based Reinforcement Learning with Isolated Imaginations](https://mlsanity.com/paper/2303.14889/) (2023)
- [Revisiting Backdoor Attacks on Time Series Classification in the Frequency Domain](https://mlsanity.com/paper/2503.09712/) (2025)
- [DaDe: Delay-adaptive Detector for Streaming Perception](https://mlsanity.com/paper/2212.11558/) (2022)
- [UniSTD: Towards Unified Spatio-Temporal Learning across Diverse Disciplines](https://mlsanity.com/paper/2503.20748/) (2025)

## Abstract (arXiv)

Video prediction is a pixel-wise dense prediction task to infer future frames based on past frames. Missing appearance details and motion blur are still two major problems for current predictive models, which lead to image distortion and temporal inconsistency. In this paper, we point out the necessity of exploring multi-frequency analysis to deal with the two problems. Inspired by the frequency band decomposition characteristic of Human Vision System (HVS), we propose a video prediction network based on multi-level wavelet analysis to deal with spatial and temporal information in a unified manner. Specifically, the multi-level spatial discrete wavelet transform decomposes each video frame into anisotropic sub-bands with multiple frequencies, helping to enrich structural information and reserve fine details. On the other hand, multi-level temporal discrete wavelet transform which operates on time axis decomposes the frame sequence into sub-band groups of different frequencies to accurately capture multi-frequency motions under a fixed frame rate. Extensive experiments on diverse datasets demonstrate that our model shows significant improvements on fidelity and temporal consistency over state-of-the-art works.