mlsanityData to 7 Oct 2026

Continuous Spatiotemporal Transformers

arXiv 2301.13338 ↗First posted 31 Jan 2023cs.LG cs.CVcited by 7

Not on the map. Why?

Abstract (arXiv)

Modeling spatiotemporal dynamical systems is a fundamental challenge in machine learning. Transformer models have been very successful in NLP and computer vision where they provide interpretable representations of data. However, a limitation of transformers in modeling continuous dynamical systems is that they are fundamentally discrete time and space models and thus have no guarantees regarding continuous sampling. To address this challenge, we present the Continuous Spatiotemporal Transformer (CST), a new transformer architecture that is designed for the modeling of continuous systems. This new framework guarantees a continuous and smooth output via optimization in Sobolev space. We benchmark CST against traditional transformers as well as other spatiotemporal dynamics modeling methods and achieve superior performance in a number of tasks on synthetic and real systems, including learning brain dynamics from calcium imaging data.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Builds on 25 of its 65 references that are in the map. First posted on arXiv 31 Jan 2023.

Builds on

25 of 65 references.

References · 25

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  4. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  5. 2018Neural Ordinary Differential Equationscited by 3,733
  6. 2020Fourier Neural Operator for Parametric Partial Differential Equationscited by 2,806
  7. 2019DeepONet: Learning nonlinear operators for identifying differential equations based on the universal approximation theorem of operatorscited by 1,965
  8. 2021ViViT: A Video Vision Transformercited by 1,610
  9. 2021Is Space-Time Attention All You Need for Video Understanding?cited by 1,609
  10. 2020Rethinking Attention with Performerscited by 1,584
  11. 2015Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcastingcited by 1,428
  12. 2020Neural Operator: Graph Kernel Network for Partial Differential Equationscited by 722
13 more
  1. 2021Nystr"omformer: A Nystr"om-Based Algorithm for Approximating Self-Attentioncited by 468
  2. 2021Neural Operator: Learning Maps Between Function Spacescited by 413
  3. 2021Choose a Transformer: Fourier or Galerkincited by 333
  4. 2016Deep Predictive Coding Networks for Video Prediction and Unsupervised Learningcited by 203
  5. 2021Pretrained Transformers as Universal Computation Enginescited by 162
  6. 2015Delving Deeper into Convolutional Networks for Learning Video Representationscited by 146
  7. 2017Sobolev Training for Neural Networkscited by 136
  8. 2017Sobolev Norm Learning Rates for Regularized Least-Squares Algorithmcited by 113
  9. 2022How to Understand Masked Autoencoderscited by 37
  10. 2020Hypersolvers: Toward Fast Continuous-Depth Modelscited by 33
  11. 2022Learning continuous models for continuous physicscited by 28
  12. 2017Recurrent computations for visual pattern completioncited by 22
  13. 2022Replacing Automatic Differentiation by Sobolev Cubatures fastens Physics Informed Neural Nets and strengthens their Approximation Powercited by 4

Built on by

Cited by year

Citing papers per year: 2023: 3; 2024: 2; 2025: 2.

7 papers.

  1. 2023Scalable Transformer for PDE Surrogate Modelingcited by 123
  2. 2023Improved Operator Learning by Orthogonal Attentioncited by 58
  3. 2025Language Models Are Implicitly Continuouscited by 4
  4. 2024STC-ViT: Spatio Temporal Continuous Vision Transformer for Medium-range Global Weather Forecastingcited by 3
  5. 2025GITO: Graph-Informed Transformer Operator for Learning Complex Partial Differential Equationscited by 2
  6. 2023Convolutional GRU Network for Seasonal Prediction of the El Ni~no-Southern Oscillationcited by 1
  7. 2024Leray-Schauder Mappings for Operator Learningcited by 1