# MATEY: multiscale adaptive foundation models for spatiotemporal physical systems

arXiv: https://arxiv.org/abs/2412.20601 · HTML: https://mlsanity.com/paper/2412.20601/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 24 references that are in the map.
- First posted on arXiv 29 Dec 2024.

## Builds on (most cited first, 11 of 11)

- [ViViT: A Video Vision Transformer](https://mlsanity.com/paper/2103.15691/) (2021)
- [Sora: A Review on Background, Technology, Limitations, and Opportunities of Large Vision Models](https://mlsanity.com/paper/2402.17177/) (2024)
- [Axial Attention in Multidimensional Transformers](https://mlsanity.com/paper/1912.12180/) (2019)
- [DeepSpeed Ulysses: System Optimizations for Enabling Training of Extreme Long Sequence Transformer Models](https://mlsanity.com/paper/2309.14509/) (2023)
- [Poseidon: Efficient Foundation Models for PDEs](https://mlsanity.com/paper/2405.19101/) (2024)
- [DPOT: Auto-Regressive Denoising Operator Transformer for Large-Scale PDE Pre-Training](https://mlsanity.com/paper/2403.03542/) (2024)
- [Multiple Physics Pretraining for Physical Surrogate Models](https://mlsanity.com/paper/2310.02994/) (2023)
- [Unisolver: PDE-Conditional Transformers Towards Universal Neural PDE Solvers](https://mlsanity.com/paper/2405.17527/) (2024)
- [ViTAR: Vision Transformer with Any Resolution](https://mlsanity.com/paper/2403.18361/) (2024)
- [A Foundation Model for the Earth System](https://mlsanity.com/paper/2405.13063/) (2024)
- [Adaptive Patching for High-resolution Image Segmentation with Transformers](https://mlsanity.com/paper/2404.09707/) (2024)

## Built on by (most cited first, 5 of 5)

- [PDE-Transformer: Efficient and Versatile Transformers for Physics Simulations](https://mlsanity.com/paper/2505.24717/) (2025)
- [Pixel-Resolved Long-Context Learning for Turbulence at Exascale: Resolving Small-scale Eddies Toward the Viscous Limit](https://mlsanity.com/paper/2507.16697/) (2025)
- [Autoregressive long-horizon prediction of plasma edge dynamics](https://mlsanity.com/paper/2512.23884/) (2025)
- [Automated Data Readiness for Scientific AI](https://mlsanity.com/paper/2607.02771/) (2026)
- [Dynamic Quadtree Tokenization and Transformer for Adaptive Mesh PDE Forecasting](https://mlsanity.com/paper/2610.04044/) (2026)

## Abstract (arXiv)

Accurate representation of the multiscale features in spatiotemporal physical systems using vision transformer (ViT) architectures requires extremely long, computationally prohibitive token sequences. To address this issue, we propose two adaptive tokenization schemes that dynamically adjust patch sizes based on local features: one ensures convergent behavior to uniform patch refinement, while the other offers better computational efficiency. Moreover, we present a set of spatiotemporal attention schemes, where the temporal or axial spatial dimensions are decoupled, and evaluate their computational and data efficiencies. We assess the performance of the proposed multiscale adaptive model, MATEY, in a sequence of experiments. The results show that adaptive tokenization schemes achieve improved accuracy without significantly increasing the length of the token sequence. Compared to a full spatiotemporal attention scheme or a scheme that decouples only the temporal dimension, we find that fully decoupled axial attention is less efficient and expressive, requiring more training time and model weights to achieve the same accuracy. Finally, we demonstrate in two fine-tuning tasks featuring different physics that models pretrained on PDEBench data outperform the ones trained from scratch, especially in the low data regime with frozen attention.