mlsanityData to 14 Oct 2026

Temporal Attention Unit: Towards Efficient Spatiotemporal Predictive Learning

arXiv 2206.12126 ↗First posted 24 Jun 2022cs.CV cs.AIcited by 86

Not on the map. Why?

Abstract (arXiv)

Spatiotemporal predictive learning aims to generate future frames by learning from historical frames. In this paper, we investigate existing methods and present a general framework of spatiotemporal predictive learning, in which the spatial encoder and decoder capture intra-frame features and the middle temporal module catches inter-frame correlations. While the mainstream methods employ recurrent units to capture long-term temporal dependencies, they suffer from low computational efficiency due to their unparallelizable architectures. To parallelize the temporal module, we propose the Temporal Attention Unit (TAU), which decomposes the temporal attention into intra-frame statical attention and inter-frame dynamical attention. Moreover, while the mean squared error loss focuses on intra-frame errors, we introduce a novel differential divergence regularization to take inter-frame variations into account. Extensive experiments demonstrate that the proposed method enables the derived model to achieve competitive performance on various spatiotemporal prediction benchmarks.

Key facts. Cited by 86 papers in the map (data through 14 Oct 2026). Builds on 79 of its 113 references that are in the map. First posted on arXiv 24 Jun 2022.

Builds on

79 of 113 references.

References · 79

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  5. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  6. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,639
  7. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  8. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  9. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,442
  10. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  11. 2015You Only Look Once: Unified, Real-Time Object Detectioncited by 4,852
  12. 2017Squeeze-and-Excitation Networkscited by 4,789
38 more
  1. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,750
  2. 2022A ConvNet for the 2020scited by 3,579
  3. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,811
  4. 2016Density estimation using Real NVPcited by 2,042
  5. 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,838
  6. 2020Rethinking Attention with Performerscited by 1,601
  7. 2018Unsupervised Representation Learning by Predicting Image Rotationscited by 1,591
  8. 2015Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcastingcited by 1,436
  9. 2016Context Encoders: Feature Learning by Inpaintingcited by 1,428
  10. 2020Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hyperspherecited by 1,405
  11. 2016Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzlescited by 1,382
  12. 2020Big Self-Supervised Models are Strong Semi-Supervised Learnerscited by 1,380
  13. 2016Colorful Image Colorizationcited by 1,114
  14. 2014NICE: Non-linear Independent Components Estimationcited by 1,058
  15. 2018Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representationscited by 941
  16. 2017Convolutional Sequence to Sequence Learningcited by 552
  17. 2019Variational Autoencoders and Nonlinear ICA: A Unifying Frameworkcited by 503
  18. 2015Unsupervised Learning of Video Representations using LSTMscited by 435
  19. 2020Spatiotemporal Contrastive Video Representation Learningcited by 372
  20. 2019Multi-Object Representation Learning with Iterative Variational Inferencecited by 360
  21. 2021Random Feature Attentioncited by 354
  22. 2016Deep Spatio-Temporal Residual Networks for Citywide Crowd Flows Predictioncited by 333
  23. 2016Dynamic Filter Networkscited by 319
  24. 2015Deep multi-scale video prediction beyond mean square errorcited by 300
  25. 2016Unsupervised Learning for Physical Interaction through Video Predictioncited by 260
  26. 2018Stochastic Video Generation with a Learned Priorcited by 245
  27. 2017Stochastic Variational Video Predictioncited by 234
  28. 2022Visual Attention Networkcited by 210
  29. 2017Video Frame Synthesis using Deep Voxel Flowcited by 209
  30. 2016Deep Predictive Coding Networks for Video Prediction and Unsupervised Learningcited by 204
  31. 2020Disentangling Physical Dynamics from Unknown Factors for Unsupervised Video Predictioncited by 195
  32. 2022SimVP: Simpler yet Better Video Predictioncited by 184
  33. 2017Deep Learning for Precipitation Nowcasting: A Benchmark and A New Modelcited by 182
  34. 2021PredRNN: A Recurrent Neural Network for Spatiotemporal Predictive Learningcited by 182
  35. 2018Stochastic Adversarial Video Predictioncited by 173
  36. 2017Decomposing Motion and Content for Natural Video Sequence Predictioncited by 169
  37. 2017Unsupervised Learning of Disentangled Representations from Videocited by 139
  38. 2021How to represent part-whole hierarchies in a neural networkcited by 123

50 most cited of 79 All 79 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 3; 2023: 14; 2024: 28; 2025: 22; 2026: 19.

86 papers.

Sort:

By area:

  1. 2024UniST: A Prompt-Empowered Universal Model for Urban Spatio-Temporal Predictioncited by 87
  2. 2022PiFold: Toward effective and efficient protein inverse foldingcited by 83
  3. 2023DiffCast: A Unified Framework via Residual Diffusion for Precipitation Nowcastingcited by 61
  4. 2023OpenSTL: A Comprehensive Benchmark of Spatio-Temporal Predictive Learningcited by 54
  5. 2023CVT-SLR: Contrastive Visual-Textual Transformation for Sign Language Recognition with Variational Alignmentcited by 45
  6. 2024VMRNN: Integrating Vision Mamba and LSTM for Efficient and Accurate Spatiotemporal Forecastingcited by 35
  7. 2023Precipitation nowcasting with generative diffusion modelscited by 23
  8. 2024Fourier Amplitude and Correlation Loss: Beyond Using L2 Loss for Skillful Precipitation Nowcastingcited by 16
  9. 2024Retrieval Meets Reasoning: Even High-school Textbook Knowledge Benefits Multimodal Reasoningcited by 14
  10. 2024SUMix: Mixup with Semantic and Uncertain Informationcited by 12
  11. 2024E-Motion: Future Motion Simulation via Event Sequence Diffusioncited by 12
  12. 2023Enhancing Human-like Multi-Modal Reasoning: A New Challenging Dataset and Comprehensive Frameworkcited by 11
38 more
  1. 2025Enhance-A-Video: Better Generated Video for Freecited by 11
  2. 2024PostCast: Generalizable Postprocessing for Precipitation Nowcasting via Unsupervised Blurriness Modelingcited by 10
  3. 2024Diffusion Transformers as Open-World Spatiotemporal Foundation Modelscited by 10
  4. 2023Deterministic Guidance Diffusion Model for Probabilistic Weather Forecastingcited by 9
  5. 2023Cross-Gate MLP with Protein Complex Invariant Embedding is A One-Shot Antibody Designercited by 8
  6. 2023PrefixMol: Target- and Chemistry-aware Molecule Design via Prefix Embeddingcited by 7
  7. 2023Triplet Attention Transformer for Spatiotemporal Predictive Learningcited by 7
  8. 2024DynST: Dynamic Sparse Training for Resource-Constrained Spatio-Temporal Forecastingcited by 7
  9. 2024UniFlow: A Foundation Model for Unified Urban Spatio-Temporal Flow Predictioncited by 7
  10. 2024PredBench: Benchmarking Spatio-Temporal Prediction across Diverse Disciplinescited by 6
  11. 2024Video Prediction Transformers without Recurrence or Convolutioncited by 6
  12. 2024Data-driven Precipitation Nowcasting Using Satellite Imagerycited by 6
  13. 2024Spatio-Temporal Fluid Dynamics Modeling via Physical-Awareness and Parameter Diffusion Guidancecited by 5
  14. 2024Advection Augmented Convolutional Neural Networkscited by 5
  15. 2025Unveiling the Power of Noise Priors: Enhancing Diffusion Models for Mobile Traffic Predictioncited by 4
  16. 2025BeamVQ: Beam Search with Vector Quantization to Mitigate Data Scarcity in Physical Spatiotemporal Forecastingcited by 4
  17. 2025Frequency-Compensated Network for Daily Arctic Sea Ice Concentration Predictioncited by 4
  18. 2025Breaking the Discretization Barrier of Continuous Physics Simulation Learningcited by 4
  19. 2022A case study of spatiotemporal forecasting techniques for weather forecastingcited by 3
  20. 2023Rethinking Urban Mobility Prediction: A Super-Multivariate Time Series Forecasting Approachcited by 3
  21. 2023Does Vector Quantization Fail in Spatio-Temporal Forecasting? Exploring a Differentiable Sparse Soft-Vector Quantization Approachcited by 3
  22. 2024BeamVQ: Aligning Space-Time Forecasting Model via Self-training on Physics-aware Metricscited by 3
  23. 2024A general reduced-order neural operator for spatio-temporal predictive learning on complex spatial domainscited by 3
  24. 2025UniSTD: Towards Unified Spatio-Temporal Learning across Diverse Disciplinescited by 3
  25. 2026Stabilizing Streaming Video Geometry via Dynamic Feature Normalizationcited by 3
  26. 2022Predictive Coding Based Multiscale Network with Encoder-Decoder LSTM for Video Predictioncited by 2
  27. 2024TAFormer: A Unified Target-Aware Transformer for Video and Motion Joint Prediction in Aerial Scenescited by 2
  28. 2024ACE Metric: Advection and Convection Evaluation for Accurate Weather Forecastingcited by 2
  29. 2024TAUDiff: Highly efficient kilometer-scale downscaling using generative diffusion modelscited by 2
  30. 2025DyMoDreamer: World Modeling with Dynamic Modulationcited by 2
  31. 2025SimCast: Enhancing Precipitation Nowcasting with Short-to-Long Term Knowledge Distillationcited by 2
  32. 2025SynCast: Synergizing Contradictions in Precipitation Nowcasting via Diffusion Sequential Preference Optimizationcited by 2
  33. 2023USTEP: Spatio-Temporal Predictive Learning under A Unified Viewcited by 1
  34. 2024M4Fog: A Global Multi-Regional, Multi-Modal, and Multi-Stage Dataset for Marine Fog Detection and Forecasting to Bridge Ocean and Atmospherecited by 1
  35. 2024STLight: a Fully Convolutional Approach for Efficient Predictive Learning by Spatio-Temporal joint Processingcited by 1
  36. 2024Federated Learning for Traffic Flow Prediction with Synthetic Data Augmentationcited by 1
  37. 2024Spatio-Temporal Multi-Subgraph GCN for 3D Human Motion Predictioncited by 1
  38. 2025RLER-TTE: An Efficient and Effective Framework for En Route Travel Time Estimation with Reinforcement Learningcited by 1

50 most cited of 86 All 86 (JSON)