mlsanityData to 7 Oct 2026

MMVP: Motion-Matrix-based Video Prediction

arXiv 2308.16154 ↗First posted 30 Aug 2023cs.CVcited by 18

Not on the map. Why?

Abstract (arXiv)

A central challenge of video prediction lies where the system has to reason the objects' future motions from image frames while simultaneously maintaining the consistency of their appearances across frames. This work introduces an end-to-end trainable two-stream video prediction framework, Motion-Matrix-based Video Prediction (MMVP), to tackle this challenge. Unlike previous methods that usually handle motion prediction and appearance maintenance within the same set of modules, MMVP decouples motion and appearance information by constructing appearance-agnostic motion matrices. The motion matrices represent the temporal similarity of each and every pair of feature patches in the input frames, and are the sole input of the motion prediction module in MMVP. This design improves video prediction in both accuracy and efficiency, and reduces the model size. Results of extensive experiments demonstrate that MMVP outperforms state-of-the-art systems on public data sets by non-negligible large margins (about 1 db in PSNR, UCF Sports) in significantly smaller model sizes (84% the size or smaller).

Key facts. Cited by 18 papers in the map (data through 7 Oct 2026). Builds on 41 of its 59 references that are in the map. First posted on arXiv 30 Aug 2023.

Builds on

41 of 59 references.

References · 41

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  3. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,029
  4. 2015Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcastingcited by 1,428
  5. 2018ESRGAN: Enhanced Super-Resolution Generative Adversarial Networkscited by 1,250
  6. 2016Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Networkcited by 1,199
  7. 2015Unsupervised Learning of Video Representations using LSTMscited by 434
  8. 2016Dynamic Filter Networkscited by 318
  9. 2015Deep multi-scale video prediction beyond mean square errorcited by 297
  10. 2016Deep Visual Foresight for Planning Robot Motioncited by 276
  11. 2017Stochastic Variational Video Predictioncited by 234
  12. 2018Visual Foresight: Model-Based Deep Reinforcement Learning for Vision-Based Robotic Controlcited by 220
29 more
  1. 2020Disentangling Physical Dynamics from Unknown Factors for Unsupervised Video Predictioncited by 192
  2. 2019Scaling Autoregressive Video Modelscited by 189
  3. 2021PredRNN: A Recurrent Neural Network for Spatiotemporal Predictive Learningcited by 181
  4. 2022SimVP: Simpler yet Better Video Predictioncited by 181
  5. 2018Stochastic Adversarial Video Predictioncited by 173
  6. 2017Decomposing Motion and Content for Natural Video Sequence Predictioncited by 168
  7. 2017Unsupervised Learning of Disentangled Representations from Videocited by 138
  8. 2018PredRNN++: Towards A Resolution of the Deep-in-Time Dilemma in Spatiotemporal Predictive Learningcited by 122
  9. 2018Learning to Decompose and Disentangle Representations for Video Predictioncited by 120
  10. 2020A Review on Deep Learning Techniques for Video Predictioncited by 114
  11. 2022MaskViT: Masked Visual Pre-Training for Video Predictioncited by 112
  12. 2016Video Pixel Networkscited by 108
  13. 2020M-LVC: Multiple Frames Prediction for Learned Video Compressioncited by 106
  14. 2020Latent Video Transformercited by 106
  15. 2018Memory In Memory: A Predictive Neural Network for Learning Higher-Order Non-Stationarity from Spatiotemporal Dynamicscited by 91
  16. 2017Dual Motion GAN for Future-Flow Embedded Video Predictioncited by 69
  17. 2019Deep Visual MPC-Policy Learning for Navigationcited by 54
  18. 2021MotionRNN: A Flexible Model for Video Prediction with Spacetime-Varying Motionscited by 54
  19. 2023A Dynamic Multi-Scale Voxel Flow Network for Video Predictioncited by 53
  20. 2020Exploring Spatial-Temporal Multi-Frequency Analysis for High-Fidelity and Temporal-Consistency Video Predictioncited by 52
  21. 2017Folded Recurrent Neural Networks for Future Video Predictioncited by 44
  22. 2018Disentangling Propagation and Generation for Video Predictioncited by 44
  23. 2022STRPM: A Spatiotemporal Residual Predictive Model for High-Resolution Video Predictioncited by 31
  24. 2022Optimizing Video Prediction via Video Frame Interpolationcited by 28
  25. 2021Learning Semantic-Aware Dynamics for Video Predictioncited by 21
  26. 2019CrevNet: Conditionally Reversible Video Predictioncited by 15
  27. 2022Advancing Learned Video Compression with In-loop Frame Predictioncited by 14
  28. 2021Accurate Grid Keypoint Learning for Efficient Video Predictioncited by 13
  29. 2022STIP: A SpatioTemporal Information-Preserving and Perception-Augmented Model for High-Resolution Video Predictioncited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 3; 2024: 5; 2025: 5; 2026: 5.

18 papers.

Sort:

By area:

  1. 2024VMRNN: Integrating Vision Mamba and LSTM for Efficient and Accurate Spatiotemporal Forecastingcited by 35
  2. 2023STDiff: Spatio-temporal Diffusion for Continuous Stochastic Video Predictioncited by 12
  3. 2023Deterministic Guidance Diffusion Model for Probabilistic Weather Forecastingcited by 9
  4. 2024A Survey on Future Frame Synthesis: Bridging Deterministic and Generative Approachescited by 7
  5. 2024Motion Graph Unleashed: A Novel Approach to Video Predictioncited by 4
  6. 2025CoMemo: LVLMs Need Image Context with Image Memorycited by 4
  7. 2023Does Vector Quantization Fail in Spatio-Temporal Forecasting? Exploring a Differentiable Sparse Soft-Vector Quantization Approachcited by 3
  8. 2025SeeU: Seeing the Unseen World via 4D Dynamics-aware Generationcited by 3
  9. 2024TAFormer: A Unified Target-Aware Transformer for Video and Motion Joint Prediction in Aerial Scenescited by 2
  10. 2025Learning Extremely High Density Crowds as Active Matterscited by 1
  11. 2026E-TIDE: Fast, Structure-Preserving Motion Forecasting from Event Sequencescited by 1
  12. 2024Denoising Reuse: Exploiting Inter-frame Motion Consistency for Efficient Video Latent Generationcited by 0
6 more
  1. 2025GMG: A Video Prediction Method Based on Global Focus and Motion Guidedcited by 0
  2. 2025Watermarking for Factuality: Guiding Vision-Language Models Toward Truth via Tri-layer Contrastive Decodingcited by 0
  3. 2026CloDS: Visual-Only Unsupervised Cloth Dynamics Learning in Unknown Conditionscited by 0
  4. 2026PFGNet: A Fully Convolutional Frequency-Guided Peripheral Gating Network for Efficient Spatiotemporal Predictive Learningcited by 0
  5. 2026WaveSFNet: A Wavelet-Based Codec and Spatial--Frequency Dual-Domain Gating Network for Spatiotemporal Predictioncited by 0
  6. 2026When the Past Matters: FlashBack Memory for Precipitation Nowcastingcited by 0