# Representation Learning for Spatiotemporal Physical Systems

arXiv: https://arxiv.org/abs/2603.13227 · HTML: https://mlsanity.com/paper/2603.13227/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 30 references that are in the map.
- First posted on arXiv 13 Mar 2026.

## Builds on (most cited first, 24 of 24)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Unsupervised Learning of Visual Features by Contrasting Cluster Assignments](https://mlsanity.com/paper/2006.09882/) (2020)
- [Improved Baselines with Momentum Contrastive Learning](https://mlsanity.com/paper/2003.04297/) (2020)
- [VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Training](https://mlsanity.com/paper/2203.12602/) (2022)
- [VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learning](https://mlsanity.com/paper/2105.04906/) (2021)
- [Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecture](https://mlsanity.com/paper/2301.08243/) (2023)
- [V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning](https://mlsanity.com/paper/2506.09985/) (2025)
- [DGM: A deep learning algorithm for solving partial differential equations](https://mlsanity.com/paper/1708.07469/) (2017)
- [The Deep Ritz method: A deep learning-based numerical algorithm for solving variational problems](https://mlsanity.com/paper/1710.00211/) (2017)
- [Solving high-dimensional partial differential equations using deep learning](https://mlsanity.com/paper/1707.02568/) (2017)
- [Revisiting Feature Prediction for Learning Visual Representations from Video](https://mlsanity.com/paper/2404.08471/) (2024)
- [Multiple Physics Pretraining for Physical Surrogate Models](https://mlsanity.com/paper/2310.02994/) (2023)
- [Towards a Foundation Model for Partial Differential Equations: Multi-Operator Learning and Extrapolation](https://mlsanity.com/paper/2404.12355/) (2024)
- [Walrus: A Cross-Domain Foundation Model for Continuum Dynamics](https://mlsanity.com/paper/2511.15684/) (2025)
- [Unsupervised Deep Learning Algorithm for PDE-based Forward and Inverse Problems](https://mlsanity.com/paper/1904.05417/) (2019)
- [Self-Supervised Learning with Lie Symmetries for Partial Differential Equations](https://mlsanity.com/paper/2307.05432/) (2023)
- [DISCO: learning to DISCover an evolution Operator for multi-physics-agnostic prediction](https://mlsanity.com/paper/2504.19496/) (2025)
- [Learning fast, accurate, and stable closures of a kinetic theory of an active fluid](https://mlsanity.com/paper/2308.06675/) (2023)
- [Predicting partially observable dynamical systems via diffusion models with a multiscale inference scheme](https://mlsanity.com/paper/2511.19390/) (2025)

## Built on by (most cited first, 11 of 11)

- [AeroJEPA: Learning Semantic Latent Representations for Scalable 3D Aerodynamic Field Modeling](https://mlsanity.com/paper/2605.05586/) (2026)
- [FluidWorld: Reaction-Diffusion Dynamics as a Predictive Substrate for World Models](https://mlsanity.com/paper/2603.21315/) (2026)
- [GLU: Global-Local-Uncertainty Fusion for Scalable Spatiotemporal Reconstruction and Forecasting](https://mlsanity.com/paper/2603.26023/) (2026)
- [StreamPhy: Streaming Inference of High-Dimensional Physical Dynamics via State Space Models](https://mlsanity.com/paper/2605.07384/) (2026)
- [Learning Spatio-Temporal Foundation Models from Pure Synthetic Data](https://mlsanity.com/paper/2607.16251/) (2026)
- [The microscope is the mask: privileged views and labels from a cryo-ET forward model](https://mlsanity.com/paper/2609.04325/) (2026)
- [RD-JEPA: Predictive latent pretraining for few-trajectory transfer across reaction--diffusion equations](https://mlsanity.com/paper/2609.29403/) (2026)
- [PDE-JEPA: Predictive Representation Learning of Latent Dynamics Modeling for Parametric PDEs](https://mlsanity.com/paper/2609.34715/) (2026)
- [Efficient Neural Field Learning via Adaptive Coverage and Focused Sampling](https://mlsanity.com/paper/2610.02410/) (2026)
- [Learning Disentangled Representations with Quantum Variational Autoencoders](https://mlsanity.com/paper/2610.07196/) (2026)
- [Cross-Domain Pretraining for Steady-State Neural CFD Surrogates](https://mlsanity.com/paper/2610.10398/) (2026)

## Abstract (arXiv)

Machine learning approaches to spatiotemporal physical systems have primarily focused on next-frame prediction, with the goal of learning an accurate emulator for the system's evolution in time. However, these emulators are computationally expensive to train and are subject to performance pitfalls, such as compounding errors during autoregressive rollout. In this work, we take a different perspective and look at scientific tasks further downstream of predicting the next frame, such as estimation of a system's governing physical parameters. Accuracy on these tasks offers a uniquely quantifiable glimpse into the physical relevance of the representations of these models. We evaluate the effectiveness of general-purpose self-supervised methods in learning physics-grounded representations that are useful for downstream scientific tasks. Surprisingly, we find that not all methods designed for physical modeling outperform generic self-supervised learning methods on these tasks, and methods that learn in the latent space (e.g., joint embedding predictive architectures, or JEPAs) outperform those optimizing pixel-level prediction objectives. Code is available at https://github.com/helenqu/physical-representation-learning.