# SwinLSTM:Improving Spatiotemporal Prediction Accuracy using Swin Transformer and LSTM

arXiv: https://arxiv.org/abs/2308.09891 · HTML: https://mlsanity.com/paper/2308.09891/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 44 references that are in the map.
- First posted on arXiv 19 Aug 2023.

## Builds on (most cited first, 30 of 30)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers](https://mlsanity.com/paper/2105.15203/) (2021)
- [Point Transformer](https://mlsanity.com/paper/2011.00931/) (2020)
- [Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting](https://mlsanity.com/paper/1506.04214/) (2015)
- [Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers](https://mlsanity.com/paper/2012.15840/) (2020)
- [Swin-Unet: Unet-like Pure Transformer for Medical Image Segmentation](https://mlsanity.com/paper/2105.05537/) (2021)
- [Understanding the Effective Receptive Field in Deep Convolutional Neural Networks](https://mlsanity.com/paper/1701.04128/) (2017)
- [Unsupervised Learning of Video Representations using LSTMs](https://mlsanity.com/paper/1502.04681/) (2015)
- [Deep Spatio-Temporal Residual Networks for Citywide Crowd Flows Prediction](https://mlsanity.com/paper/1610.00081/) (2016)
- [Dynamic Filter Networks](https://mlsanity.com/paper/1605.09673/) (2016)
- [Scaling Local Self-Attention for Parameter Efficient Visual Backbones](https://mlsanity.com/paper/2103.12731/) (2021)
- [Multi-Scale Vision Longformer: A New Vision Transformer for High-Resolution Image Encoding](https://mlsanity.com/paper/2103.15358/) (2021)
- [Disentangling Physical Dynamics from Unknown Factors for Unsupervised Video Prediction](https://mlsanity.com/paper/2003.01460/) (2020)
- [Deep Learning for Precipitation Nowcasting: A Benchmark and A New Model](https://mlsanity.com/paper/1706.03458/) (2017)
- [Stochastic Adversarial Video Prediction](https://mlsanity.com/paper/1804.01523/) (2018)
- [Decomposing Motion and Content for Natural Video Sequence Prediction](https://mlsanity.com/paper/1706.08033/) (2017)
- [A2-Nets: Double Attention Networks](https://mlsanity.com/paper/1810.11579/) (2018)
- [PredRNN++: Towards A Resolution of the Deep-in-Time Dilemma in Spatiotemporal Predictive Learning](https://mlsanity.com/paper/1804.06300/) (2018)
- [Video Pixel Networks](https://mlsanity.com/paper/1610.00527/) (2016)
- [Memory In Memory: A Predictive Neural Network for Learning Higher-Order Non-Stationarity from Spatiotemporal Dynamics](https://mlsanity.com/paper/1811.07490/) (2018)
- [P2T: Pyramid Pooling Transformer for Scene Understanding](https://mlsanity.com/paper/2106.12011/) (2021)
- [Long-Term On-Board Prediction of People in Traffic Scenes under Uncertainty](https://mlsanity.com/paper/1711.09026/) (2017)
- [Video Prediction Recalling Long-term Motion Context via Memory Alignment Learning](https://mlsanity.com/paper/2104.00924/) (2021)
- [Exploring Spatial-Temporal Multi-Frequency Analysis for High-Fidelity and Temporal-Consistency Video Prediction](https://mlsanity.com/paper/2002.09905/) (2020)
- [Folded Recurrent Neural Networks for Future Video Prediction](https://mlsanity.com/paper/1712.00311/) (2017)
- [Disentangling Propagation and Generation for Video Prediction](https://mlsanity.com/paper/1812.00452/) (2018)

## Built on by (most cited first, 30 of 32)

- [VMRNN: Integrating Vision Mamba and LSTM for Efficient and Accurate Spatiotemporal Forecasting](https://mlsanity.com/paper/2403.16536/) (2024)
- [Foundation Models for Weather and Climate Data Understanding: A Comprehensive Survey](https://mlsanity.com/paper/2312.03014/) (2023)
- [Wind speed super-resolution and validation: from ERA5 to CERRA via diffusion models](https://mlsanity.com/paper/2401.15469/) (2024)
- [Triplet Attention Transformer for Spatiotemporal Predictive Learning](https://mlsanity.com/paper/2310.18698/) (2023)
- [LITE: Modeling Environmental Ecosystems with Multimodal Large Language Models](https://mlsanity.com/paper/2404.01165/) (2024)
- [Video Prediction Transformers without Recurrence or Convolution](https://mlsanity.com/paper/2410.04733/) (2024)
- [Data-driven Precipitation Nowcasting Using Satellite Imagery](https://mlsanity.com/paper/2412.11480/) (2024)
- [Advection Augmented Convolutional Neural Networks](https://mlsanity.com/paper/2406.19253/) (2024)
- [Online Generic Event Boundary Detection](https://mlsanity.com/paper/2510.06855/) (2025)
- [Rethinking Urban Mobility Prediction: A Super-Multivariate Time Series Forecasting Approach](https://mlsanity.com/paper/2312.01699/) (2023)
- [Learning to Refine: Spectral-Decoupled Iterative Refinement Framework for Precipitation Nowcasting](https://mlsanity.com/paper/2606.02661/) (2026)
- [Adaptive Runge-Kutta Dynamics for Spatiotemporal Prediction](https://mlsanity.com/paper/2405.14504/) (2024)
- [FUME: Fused Unified Multi-Gas Emission Network for Livestock Rumen Acidosis Detection](https://mlsanity.com/paper/2601.08205/) (2026)
- [E-TIDE: Fast, Structure-Preserving Motion Forecasting from Event Sequences](https://mlsanity.com/paper/2603.27757/) (2026)
- [SIAM: A Simple Alternating Mixer for Video Prediction](https://mlsanity.com/paper/2311.11683/) (2023)
- [Resolving Spatio-Temporal Entanglement in Video Prediction via Multi-Modal Attention](https://mlsanity.com/paper/2501.16997/) (2025)
- [GMG: A Video Prediction Method Based on Global Focus and Motion Guided](https://mlsanity.com/paper/2503.11297/) (2025)
- [Deformable Dynamic Convolution for Accurate yet Efficient Spatio-Temporal Traffic Prediction](https://mlsanity.com/paper/2507.11550/) (2025)
- [HyperTea: A Hypergraph-based Temporal Enhancement and Alignment Network for Moving Infrared Small Target Detection](https://mlsanity.com/paper/2508.10678/) (2025)
- [REE-TTT: Highly Adaptive Radar Echo Extrapolation Based on Test-Time Training](https://mlsanity.com/paper/2601.01605/) (2026)
- [CloDS: Visual-Only Unsupervised Cloth Dynamics Learning in Unknown Conditions](https://mlsanity.com/paper/2602.01844/) (2026)
- [Frequency-Enhanced Hilbert Scanning Mamba for Short-Term Arctic Sea Ice Concentration Prediction](https://mlsanity.com/paper/2602.13522/) (2026)
- [PFGNet: A Fully Convolutional Frequency-Guided Peripheral Gating Network for Efficient Spatiotemporal Predictive Learning](https://mlsanity.com/paper/2602.20537/) (2026)
- [WaveSFNet: A Wavelet-Based Codec and Spatial--Frequency Dual-Domain Gating Network for Spatiotemporal Prediction](https://mlsanity.com/paper/2603.23284/) (2026)
- [Long-horizon prediction of three-dimensional wall-bounded turbulence with CTA-Swin-UNet and resolvent analysis](https://mlsanity.com/paper/2605.17888/) (2026)
- [When the Past Matters: FlashBack Memory for Precipitation Nowcasting](https://mlsanity.com/paper/2606.16342/) (2026)
- [Learning Video Dynamics with Predictive Differentiable Rendering](https://mlsanity.com/paper/2606.31050/) (2026)
- [KOALA: Koopman Operator Learning for WiFi-Based Anticipatory Hum](https://mlsanity.com/paper/2608.15815/) (2026)
- [Physics-Knowledge-Guided Hybrid Neural Learning for Arctic Sea Ice Concentration Evolution and Short-Range Prediction](https://mlsanity.com/paper/2608.21767/) (2026)
- [What Will This Copper Look Like Later? Forecasting Surface Appearance and Rendering It as a PBR Material](https://mlsanity.com/paper/2608.28102/) (2026)

## Abstract (arXiv)

Integrating CNNs and RNNs to capture spatiotemporal dependencies is a prevalent strategy for spatiotemporal prediction tasks. However, the property of CNNs to learn local spatial information decreases their efficiency in capturing spatiotemporal dependencies, thereby limiting their prediction accuracy. In this paper, we propose a new recurrent cell, SwinLSTM, which integrates Swin Transformer blocks and the simplified LSTM, an extension that replaces the convolutional structure in ConvLSTM with the self-attention mechanism. Furthermore, we construct a network with SwinLSTM cell as the core for spatiotemporal prediction. Without using unique tricks, SwinLSTM outperforms state-of-the-art methods on Moving MNIST, Human3.6m, TaxiBJ, and KTH datasets. In particular, it exhibits a significant improvement in prediction accuracy compared to ConvLSTM. Our competitive experimental results demonstrate that learning global spatial dependencies is more advantageous for models to capture spatiotemporal dependencies. We hope that SwinLSTM can serve as a solid baseline to promote the advancement of spatiotemporal prediction accuracy. The codes are publicly available at https://github.com/SongTang-x/SwinLSTM.