STAU: A SpatioTemporal-Aware Unit for Video Prediction and Beyond
Not on the map. Why?
Abstract (arXiv)
Video prediction aims to predict future frames by modeling the complex spatiotemporal dynamics in videos. However, most of the existing methods only model the temporal information and the spatial information for videos in an independent manner but haven't fully explored the correlations between both terms. In this paper, we propose a SpatioTemporal-Aware Unit (STAU) for video prediction and beyond by exploring the significant spatiotemporal correlations in videos. On the one hand, the motion-aware attention weights are learned from the spatial states to help aggregate the temporal states in the temporal domain. On the other hand, the appearance-aware attention weights are learned from the temporal states to help aggregate the spatial states in the spatial domain. In this way, the temporal information and the spatial information can be greatly aware of each other in both domains, during which, the spatiotemporal receptive field can also be greatly broadened for more reliable spatiotemporal modeling. Experiments are not only conducted on traditional video prediction tasks but also other tasks beyond video prediction, including the early action recognition and object detection tasks. Experimental results show that our STAU can outperform other methods on all tasks in terms of performance and computation efficiency.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 36 of its 66 references that are in the map. First posted on arXiv 20 Apr 2022.
Builds on
36 of 66 references.
References · 36
- 2014Adam: A Method for Stochastic Optimizationcited by 36,960
- 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,029
- 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
- 2015You Only Look Once: Unified, Real-Time Object Detectioncited by 4,829
- 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,016
- 2015Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcastingcited by 1,428
- 2018Towards Accurate Generative Models of Video: A New Metric & Challengescited by 1,204
- 2017The "something something" video database for learning and evaluating visual common sensecited by 1,092
- 2015Unsupervised Learning of Video Representations using LSTMscited by 434
- 2015Deep multi-scale video prediction beyond mean square errorcited by 297
- 2016Unsupervised Learning for Physical Interaction through Video Predictioncited by 256
- 2018Stochastic Video Generation with a Learned Priorcited by 244
24 more
- 2017Stochastic Variational Video Predictioncited by 234
- 2016Deep Predictive Coding Networks for Video Prediction and Unsupervised Learningcited by 203
- 2020Disentangling Physical Dynamics from Unknown Factors for Unsupervised Video Predictioncited by 192
- 2017Deep Learning for Precipitation Nowcasting: A Benchmark and A New Modelcited by 180
- 2018Stochastic Adversarial Video Predictioncited by 173
- 2017Decomposing Motion and Content for Natural Video Sequence Predictioncited by 168
- 2015Delving Deeper into Convolutional Networks for Learning Video Representationscited by 146
- 2017Self-Supervised Visual Planning with Temporal Skip Connectionscited by 137
- 2018PredRNN++: Towards A Resolution of the Deep-in-Time Dilemma in Spatiotemporal Predictive Learningcited by 122
- 2020A Review on Deep Learning Techniques for Video Predictioncited by 114
- 2016Video Pixel Networkscited by 108
- 2020Stochastic Latent Residual Video Predictioncited by 106
- 2014Video (language) modeling: a baseline for generative models of natural videoscited by 92
- 2018Memory In Memory: A Predictive Neural Network for Learning Higher-Order Non-Stationarity from Spatiotemporal Dynamicscited by 91
- 2019Image and Video Compression with Neural Networks: A Reviewcited by 82
- 2021Greedy Hierarchical Variational Autoencoders for Large-Scale Video Predictioncited by 75
- 2020Transformation-based Adversarial Video Prediction on Large-Scale Datacited by 66
- 2021Video Prediction Recalling Long-term Motion Context via Memory Alignment Learningcited by 62
- 2021MotionRNN: A Flexible Model for Video Prediction with Spacetime-Varying Motionscited by 54
- 2020Exploring Spatial-Temporal Multi-Frequency Analysis for High-Fidelity and Temporal-Consistency Video Predictioncited by 52
- 2020Probabilistic Future Prediction for Video Scene Understandingcited by 45
- 2017Folded Recurrent Neural Networks for Future Video Predictioncited by 44
- 2020Video Prediction via Example Guidancecited by 9
- 2021CMS-LSTM: Context Embedding and Multi-Scale Spatiotemporal Expression LSTM for Predictive Learningcited by 4
Built on by
Citing papers per year: 2025: 1; 2026: 1.
2 papers.