# Deciphering and integrating invariants for neural operator learning with various physical mechanisms

arXiv: https://arxiv.org/abs/2311.14361 · HTML: https://mlsanity.com/paper/2311.14361/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 54 references that are in the map.
- First posted on arXiv 24 Nov 2023.

## Builds on (most cited first, 25 of 25)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Fourier Neural Operator for Parametric Partial Differential Equations](https://mlsanity.com/paper/2010.08895/) (2020)
- [DeepONet: Learning nonlinear operators for identifying differential equations based on the universal approximation theorem of operators](https://mlsanity.com/paper/1910.03193/) (2019)
- [Solving high-dimensional partial differential equations using deep learning](https://mlsanity.com/paper/1707.02568/) (2017)
- [PDEBENCH: An Extensive Benchmark for Scientific Machine Learning](https://mlsanity.com/paper/2210.07182/) (2022)
- [Message Passing Neural PDE Solvers](https://mlsanity.com/paper/2202.03376/) (2022)
- [Spatiotemporal Contrastive Video Representation Learning](https://mlsanity.com/paper/2008.03800/) (2020)
- [Dynamic Convolution: Attention over Convolution Kernels](https://mlsanity.com/paper/1912.03458/) (2019)
- [Choose a Transformer: Fourier or Galerkin](https://mlsanity.com/paper/2105.14995/) (2021)
- [Transformer for Partial Differential Equations' Operator Learning](https://mlsanity.com/paper/2205.13671/) (2022)
- [Factorized Fourier Neural Operators](https://mlsanity.com/paper/2111.13802/) (2021)
- [Multiwavelet-based Operator Learning for Differential Equations](https://mlsanity.com/paper/2109.13459/) (2021)
- [U-NO: U-shaped Neural Operators](https://mlsanity.com/paper/2204.11127/) (2022)
- [Physics-informed learning of governing equations from scarce data](https://mlsanity.com/paper/2005.03448/) (2020)
- [VideoMoCo: Contrastive Video Representation Learning with Temporally Adversarial Examples](https://mlsanity.com/paper/2103.05905/) (2021)
- [A comprehensive deep learning-based approach to reduced order modeling of nonlinear time-dependent parametrized PDEs](https://mlsanity.com/paper/2001.04001/) (2020)
- [NOMAD: Nonlinear Manifold Decoders for Operator Learning](https://mlsanity.com/paper/2206.03551/) (2022)
- [Neural Inverse Operators for Solving PDE Inverse Problems](https://mlsanity.com/paper/2301.11167/) (2023)
- [Meta-Learning Dynamics Forecasting Using Task Inference](https://mlsanity.com/paper/2102.10271/) (2021)
- [SVD Perspectives for Augmenting DeepONet Flexibility and Interpretability](https://mlsanity.com/paper/2204.12670/) (2022)
- [SCVRL: Shuffled Contrastive Video Representation Learning](https://mlsanity.com/paper/2205.11710/) (2022)
- [NeuralStagger: Accelerating Physics-constrained Neural PDE Solver with Spatial-temporal Decomposition](https://mlsanity.com/paper/2302.10255/) (2023)
- [Deep Random Vortex Method for Simulation and Inference of Navier-Stokes Equations](https://mlsanity.com/paper/2206.09571/) (2022)

## Built on by (most cited first, 15 of 15)

- [DISCO: learning to DISCover an evolution Operator for multi-physics-agnostic prediction](https://mlsanity.com/paper/2504.19496/) (2025)
- [Masked Autoencoders are PDE Learners](https://mlsanity.com/paper/2403.17728/) (2024)
- [Strategies for Pretraining Neural Operators](https://mlsanity.com/paper/2406.08473/) (2024)
- [PICL: Physics Informed Contrastive Learning for Partial Differential Equations](https://mlsanity.com/paper/2401.16327/) (2024)
- [Pretraining a Neural Operator in Lower Dimensions](https://mlsanity.com/paper/2407.17616/) (2024)
- [PeSANet: Physics-encoded Spectral Attention Network for Simulating PDE-Governed Complex Systems](https://mlsanity.com/paper/2505.01736/) (2025)
- [Disentangled Representation Learning for Parametric Partial Differential Equations](https://mlsanity.com/paper/2410.02136/) (2024)
- [Geometry-Aware Neural Optimizer for Shape Optimization and Inversion](https://mlsanity.com/paper/2605.04474/) (2026)
- [PIMRL: Physics-Informed Multi-Scale Recurrent Learning for Burst-Sampled Spatiotemporal Dynamics](https://mlsanity.com/paper/2503.10253/) (2025)
- [Spectral-inspired Operator Learning with Limited Data and Unknown Physics](https://mlsanity.com/paper/2505.21573/) (2025)
- [Localized PCA-Net Neural Operators for Scalable Solution Reconstruction of Elliptic PDEs](https://mlsanity.com/paper/2509.18110/) (2025)
- [Stable spectral neural operator for learning stiff PDE systems from limited data](https://mlsanity.com/paper/2512.11686/) (2025)
- [From Complex Dynamics to DynFormer: Rethinking Transformers for PDEs](https://mlsanity.com/paper/2603.03112/) (2026)
- [Multi-Head Residual-Gated DeepONet for Coherent Nonlinear Wave Dynamics](https://mlsanity.com/paper/2604.11972/) (2026)
- [PerFlow: Physics-Embedded Rectified Flow for Efficient Reconstruction and Uncertainty Quantification of Spatiotemporal Dynamics](https://mlsanity.com/paper/2605.03548/) (2026)

## Abstract (arXiv)

Neural operators have been explored as surrogate models for simulating physical systems to overcome the limitations of traditional partial differential equation (PDE) solvers. However, most existing operator learning methods assume that the data originate from a single physical mechanism, limiting their applicability and performance in more realistic scenarios. To this end, we propose Physical Invariant Attention Neural Operator (PIANO) to decipher and integrate the physical invariants (PI) for operator learning from the PDE series with various physical mechanisms. PIANO employs self-supervised learning to extract physical knowledge and attention mechanisms to integrate them into dynamic convolutional layers. Compared to existing techniques, PIANO can reduce the relative error by 13.6%-82.2% on PDE forecasting tasks across varying coefficients, forces, or boundary conditions. Additionally, varied downstream tasks reveal that the PI embeddings deciphered by PIANO align well with the underlying invariants in the PDE systems, verifying the physical significance of PIANO. The source code will be publicly available at: https://github.com/optray/PIANO.