# Physics-informed Temporal Alignment for Auto-regressive PDE Foundation Models

arXiv: https://arxiv.org/abs/2505.10930 · HTML: https://mlsanity.com/paper/2505.10930/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 68 of its 114 references that are in the map.
- First posted on arXiv 16 May 2025.

## Builds on (most cited first, 30 of 68)

- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Fourier Neural Operator for Parametric Partial Differential Equations](https://mlsanity.com/paper/2010.08895/) (2020)
- [Shortcut Learning in Deep Neural Networks](https://mlsanity.com/paper/2004.07780/) (2020)
- [ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness](https://mlsanity.com/paper/1811.12231/) (2018)
- [Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semantics](https://mlsanity.com/paper/1705.07115/) (2017)
- [A Comprehensive Survey on Transfer Learning](https://mlsanity.com/paper/1911.02685/) (2019)
- [GradNorm: Gradient Normalization for Adaptive Loss Balancing in Deep Multitask Networks](https://mlsanity.com/paper/1711.02257/) (2017)
- [Multi-Task Learning as Multi-Objective Optimization](https://mlsanity.com/paper/1810.04650/) (2018)
- [VILA: On Pre-training for Visual Language Models](https://mlsanity.com/paper/2312.07533/) (2023)
- [Scientific Machine Learning through Physics-Informed Neural Networks: Where we are and What's next](https://mlsanity.com/paper/2201.05624/) (2022)
- [Fourier Neural Operator with Learned Deformations for PDEs on General Geometries](https://mlsanity.com/paper/2207.05209/) (2022)
- [PDEBENCH: An Extensive Benchmark for Scientific Machine Learning](https://mlsanity.com/paper/2210.07182/) (2022)
- [GNOT: A General Neural Operator Transformer for Operator Learning](https://mlsanity.com/paper/2302.14376/) (2023)
- [B-PINNs: Bayesian Physics-Informed Neural Networks for Forward and Inverse PDE Problems with Noisy Data](https://mlsanity.com/paper/2003.06097/) (2020)
- [PDE-Net: Learning PDEs from Data](https://mlsanity.com/paper/1710.09668/) (2017)
- [Transformer for Partial Differential Equations' Operator Learning](https://mlsanity.com/paper/2205.13671/) (2022)
- [Adaptive Fourier Neural Operators: Efficient Token Mixers for Transformers](https://mlsanity.com/paper/2111.13587/) (2021)
- [Model Sparsity Can Simplify Machine Unlearning](https://mlsanity.com/paper/2304.04934/) (2023)
- [Towards Multi-spatiotemporal-scale Generalized PDE Modeling](https://mlsanity.com/paper/2209.15616/) (2022)
- [Adan: Adaptive Nesterov Momentum Algorithm for Faster Optimizing Deep Models](https://mlsanity.com/paper/2208.06677/) (2022)
- [Physics-informed learning of governing equations from scarce data](https://mlsanity.com/paper/2005.03448/) (2020)
- [DPOT: Auto-Regressive Denoising Operator Transformer for Large-Scale PDE Pre-Training](https://mlsanity.com/paper/2403.03542/) (2024)
- [Ensemble-SINDy: Robust sparse model discovery in the low-data, high-noise limit, with active learning and control](https://mlsanity.com/paper/2111.10992/) (2021)
- [Towards Foundation Models for Scientific Machine Learning: Characterizing Scaling and Transfer Behavior](https://mlsanity.com/paper/2306.00258/) (2023)
- [Modeling the Dynamics of PDE Systems with Physics-Constrained Deep Auto-Regressive Networks](https://mlsanity.com/paper/1906.05747/) (2019)
- [Scalable Transformer for PDE Surrogate Modeling](https://mlsanity.com/paper/2305.17560/) (2023)
- [SINDy-PI: A Robust Algorithm for Parallel Implicit Sparse Identification of Nonlinear Dynamics](https://mlsanity.com/paper/2004.02322/) (2020)
- [Can contrastive learning avoid shortcut solutions?](https://mlsanity.com/paper/2106.11230/) (2021)
- [Auxiliary Tasks in Multi-task Learning](https://mlsanity.com/paper/1805.06334/) (2018)
- [In-Context Operator Learning with Data Prompts for Differential Equation Problems](https://mlsanity.com/paper/2304.07993/) (2023)

## Built on by (most cited first, 5 of 5)

- [Unsupervised Adaptation of PDE Foundation Models](https://mlsanity.com/paper/2608.07053/) (2026)
- [Excluding the Target Domain Improves Extrapolation: Deconfounded Hierarchical Physics Constraints](https://mlsanity.com/paper/2605.07485/) (2026)
- [ARC-STAR: Auditable Post-Hoc Correction for PDE Foundation Models](https://mlsanity.com/paper/2605.22222/) (2026)
- [MeshTok: Efficient Multi-Scale Tokenization for Scalable PDE Transformers](https://mlsanity.com/paper/2606.04366/) (2026)
- [On the post-hoc Evaluation of PDE Discovery: A Multifaceted Challenge of Scientific Advancement](https://mlsanity.com/paper/2607.23753/) (2026)

## Abstract (arXiv)

Auto-regressive partial differential equation (PDE) foundation models have shown great potential in handling time-dependent data. However, these models suffer from the shortcut problem deeply rooted in auto-regressive prediction, causing error accumulation. The challenge becomes particularly evident for out-of-distribution data, as the pretraining performance may approach random model initialization for downstream tasks with long-term dynamics. To deal with this problem, we propose physics-informed temporal alignment (PITA), a self-supervised learning framework inspired by inverse problem solving. Specifically, PITA aligns the physical dynamics discovered at different time steps on each given PDE trajectory by integrating physics-informed constraints into the self-supervision signal. The alignment is derived from observation data without relying on known physics priors, indicating strong generalization ability to the out-of-distribution data. Extensive experiments show that PITA significantly enhances the accuracy and robustness of existing foundation models on diverse time-dependent PDE data. The code is available at https://github.com/SCAILab-USTC/PITA.