mlsanityData to 7 Oct 2026

VMRNN: Integrating Vision Mamba and LSTM for Efficient and Accurate Spatiotemporal Forecasting

arXiv 2403.16536 ↗First posted 25 Mar 2024cs.CVcited by 35

On the map: State-space sequence models (S4, Mamba) → Video, motion and generative Mamba models

Abstract (arXiv)

Combining CNNs or ViTs, with RNNs for spatiotemporal forecasting, has yielded unparalleled results in predicting temporal and spatial dynamics. However, modeling extensive global information remains a formidable challenge; CNNs are limited by their narrow receptive fields, and ViTs struggle with the intensive computational demands of their attention mechanisms. The emergence of recent Mamba-based architectures has been met with enthusiasm for their exceptional long-sequence modeling capabilities, surpassing established vision models in efficiency and accuracy, which motivates us to develop an innovative architecture tailored for spatiotemporal forecasting. In this paper, we propose the VMRNN cell, a new recurrent unit that integrates the strengths of Vision Mamba blocks with LSTM. We construct a network centered on VMRNN cells to tackle spatiotemporal prediction tasks effectively. Our extensive evaluations show that our proposed approach secures competitive results on a variety of tasks while maintaining a smaller model size. Our code is available at https://github.com/yyyujintang/VMRNN-PyTorch.

Key facts. Cited by 35 papers in the map (data through 7 Oct 2026). Placed in Video, motion and generative Mamba models, one of 10 directions in State-space sequence models (S4, Mamba). Builds on 52 of its 66 references that are in the map. First posted on arXiv 25 Mar 2024.

Builds on

52 of 66 references.

References · 52

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  4. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
  5. 2021Efficiently Modeling Long Sequences with Structured State Spacescited by 2,424
  6. 2020GLU Variants Improve Transformercited by 1,724
  7. 2015Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcastingcited by 1,428
  8. 2021Swin-Unet: Unet-like Pure Transformer for Medical Image Segmentationcited by 1,218
  9. 2024Vision Mamba: Efficient Visual Representation Learning with Bidirectional State Space Modelcited by 1,051
  10. 2024VMamba: Visual State Space Modelcited by 955
  11. 2022Simplified State Space Layers for Sequence Modelingcited by 776
  12. 2021Combining Recurrent, Convolutional, and Continuous-time Models with Linear State-Space Layerscited by 607
38 more
  1. 2020HiPPO: Recurrent Memory with Optimal Polynomial Projectionscited by 600
  2. 2017Understanding the Effective Receptive Field in Deep Convolutional Neural Networkscited by 549
  3. 2022Hungry Hungry Hippos: Towards Language Modeling with State Space Modelscited by 495
  4. 2015Unsupervised Learning of Video Representations using LSTMscited by 434
  5. 2022On the Parameterization and Initialization of Diagonal State Space Modelscited by 418
  6. 2024U-Mamba: Enhancing Long-range Dependency for Biomedical Image Segmentationcited by 412
  7. 2020Spatiotemporal Contrastive Video Representation Learningcited by 372
  8. 2022Diagonal State Spaces are as Effective as Structured State Spacescited by 350
  9. 2024MambaIR: A Simple Baseline for Image Restoration with State-Space Modelcited by 349
  10. 2016Deep Spatio-Temporal Residual Networks for Citywide Crowd Flows Predictioncited by 330
  11. 2016Dynamic Filter Networkscited by 318
  12. 2024SegMamba: Long-range Sequential Modeling Mamba For 3D Medical Image Segmentationcited by 266
  13. 2022Long Range Language Modeling via Gated State Spacescited by 265
  14. 2016Deep Predictive Coding Networks for Video Prediction and Unsupervised Learningcited by 203
  15. 2020Disentangling Physical Dynamics from Unknown Factors for Unsupervised Video Predictioncited by 192
  16. 2021PredRNN: A Recurrent Neural Network for Spatiotemporal Predictive Learningcited by 181
  17. 2022SimVP: Simpler yet Better Video Predictioncited by 181
  18. 2017Deep Learning for Precipitation Nowcasting: A Benchmark and A New Modelcited by 180
  19. 2024Swin-UMamba: Mamba-based UNet with ImageNet-based pretrainingcited by 179
  20. 2022Mega: Moving Average Equipped Gated Attentioncited by 178
  21. 2018Stochastic Adversarial Video Predictioncited by 173
  22. 2022Liquid Structural State-Space Modelscited by 134
  23. 2018PredRNN++: Towards A Resolution of the Deep-in-Time Dilemma in Spatiotemporal Predictive Learningcited by 122
  24. 2016Video Pixel Networkscited by 108
  25. 2018Memory In Memory: A Predictive Neural Network for Learning Higher-Order Non-Stationarity from Spatiotemporal Dynamicscited by 91
  26. 2022Temporal Attention Unit: Towards Efficient Spatiotemporal Predictive Learningcited by 84
  27. 2020Video Representation Learning by Recognizing Temporal Transformationscited by 82
  28. 2017Long-Term On-Board Prediction of People in Traffic Scenes under Uncertaintycited by 63
  29. 2021Video Prediction Recalling Long-term Motion Context via Memory Alignment Learningcited by 62
  30. 2023A Dynamic Multi-Scale Voxel Flow Network for Video Predictioncited by 53
  31. 2023OpenSTL: A Comprehensive Benchmark of Spatio-Temporal Predictive Learningcited by 53
  32. 2020Exploring Spatial-Temporal Multi-Frequency Analysis for High-Fidelity and Temporal-Consistency Video Predictioncited by 52
  33. 2024nnMamba: 3D Biomedical Image Segmentation, Classification and Landmark Detection with State Space Modelcited by 45
  34. 2017Folded Recurrent Neural Networks for Future Video Predictioncited by 44
  35. 2018Disentangling Propagation and Generation for Video Predictioncited by 44
  36. 2017RGB-D-based Human Motion Recognition with Deep Learning: A Surveycited by 37
  37. 2023SwinLSTM:Improving Spatiotemporal Prediction Accuracy using Swin Transformer and LSTMcited by 32
  38. 2023MMVP: Motion-Matrix-based Video Predictioncited by 18

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 10; 2025: 13; 2026: 12.

35 papers.

Sort:

By area:

  1. 2024State Space Model for New-Generation Network Alternative to Transformers: A Surveycited by 59
  2. 2024A Survey on Visual Mambacited by 50
  3. 2024A Survey of Mambacited by 39
  4. 2024QMambaBSR: Burst Image Super-Resolution with Query State Space Modelcited by 27
  5. 2025Directing Mamba to Complex Textures: An Efficient Texture-Aware State Space Model for Image Restorationcited by 27
  6. 2025How Can Time Series Analysis Benefit From Multiple Modalities? A Survey and Outlookcited by 20
  7. 2024DMM: Disparity-guided Multispectral Mamba for Oriented Object Detection in Remote Sensingcited by 19
  8. 2024A Comprehensive Survey of Mamba Architectures for Medical Image Analysis: Classification, Segmentation, Restoration and Beyondcited by 16
  9. 2025Deep Learning and Foundation Models for Weather Prediction: A Surveycited by 15
  10. 2025Exploiting Temporal State Space Sharing for Video Semantic Segmentationcited by 15
  11. 2024S3Mamba: Arbitrary-Scale Super-Resolution via Scaleable State Space Modelcited by 12
  12. 2024Mamba-FSCIL: Dynamic Adaptation with Selective State Space Model for Few-Shot Class-Incremental Learningcited by 7
23 more
  1. 2024Video Prediction Transformers without Recurrence or Convolutioncited by 6
  2. 2024CLIP-Mamba: CLIP Pretrained Mamba Models with OOD and Hessian Evaluationcited by 3
  3. 2025Technologies on Effectiveness and Efficiency: A Survey of State Spaces Modelscited by 2
  4. 2025Image Recognition with Online Lightweight Vision Transformer: A Surveycited by 2
  5. 2025VMRA-MaR: An Asymmetry-Aware Temporal Framework for Longitudinal Breast Cancer Risk Predictioncited by 2
  6. 2026KFTD: Koopman-Fourier Time-Differentiable Network for Continuous Ocean Spatiotemporal Forecastingcited by 2
  7. 2025A Spatial-temporal Deep Probabilistic Diffusion Model for Reliable Hail Nowcasting with Radar Echo Extrapolationcited by 1
  8. 2025HiSTM: Hierarchical Spatiotemporal Mamba for Cellular Traffic Forecastingcited by 1
  9. 2025GMG: A Video Prediction Method Based on Global Focus and Motion Guidedcited by 0
  10. 2025QuarterMap: Efficient Post-Training Token Pruning for Visual State Space Modelscited by 0
  11. 2025SSTODE: Ocean-Atmosphere Physics-Informed Neural ODEs for Sea Surface Temperature Predictioncited by 0
  12. 2025TSkel-Mamba: Temporal Dynamic Modeling via State Space Model for Human Skeleton-based Action Recognitioncited by 0
  13. 2026REE-TTT: Highly Adaptive Radar Echo Extrapolation Based on Test-Time Trainingcited by 0
  14. 2026Frequency-Enhanced Hilbert Scanning Mamba for Short-Term Arctic Sea Ice Concentration Predictioncited by 0
  15. 2026PFGNet: A Fully Convolutional Frequency-Guided Peripheral Gating Network for Efficient Spatiotemporal Predictive Learningcited by 0
  16. 2026Spatial PDE-aware Selective State-space with Nested Memory for Mobile Traffic Grid Forecastingcited by 0
  17. 2026WaveSFNet: A Wavelet-Based Codec and Spatial--Frequency Dual-Domain Gating Network for Spatiotemporal Predictioncited by 0
  18. 2026HST-HGN: Heterogeneous Spatial-Temporal Hypergraph Networks with Bidirectional State Space Models for Global Fatigue Assessmentcited by 0
  19. 2026When the Past Matters: FlashBack Memory for Precipitation Nowcastingcited by 0
  20. 2026Learning Video Dynamics with Predictive Differentiable Renderingcited by 0
  21. 2026KOALA: Koopman Operator Learning for WiFi-Based Anticipatory Humcited by 0
  22. 2026Physics-Knowledge-Guided Hybrid Neural Learning for Arctic Sea Ice Concentration Evolution and Short-Range Predictioncited by 0
  23. 2026Structured Neural Modeling of Daily Arctic Sea-Ice Concentration Evolution: Physical-Trajectory-Driven Learning and Forecast-Domain Adaptationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025VFMF: World Modeling by Forecasting Vision Foundation Model Features
  2. 2023Convolutional State Space Models for Long-Range Spatiotemporal Modeling
  3. 2024LocalMamba: Visual State Space Model with Windowed Selective Scan
  4. 2025VCMamba: Bridging Convolutions with Multi-Directional Mamba for Efficient Visual Representation
  5. 2024SiMBA: Simplified Mamba-Based Architecture for Vision and Multivariate Time series
  6. 2026A Frame is Worth One Token: Efficient Generative World Modeling with Delta Tokens
  7. 2025STM3: Mixture of Multiscale Mamba for Long-Term Spatio-Temporal Time-Series Prediction
  8. 2024Autoregressive Pretraining with Mamba in Vision