# Self-Supervised Learning with Lie Symmetries for Partial Differential Equations

arXiv: https://arxiv.org/abs/2307.05432 · HTML: https://mlsanity.com/paper/2307.05432/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- Builds on 58 of its 105 references that are in the map.
- First posted on arXiv 11 Jul 2023.

## Builds on (most cited first, 30 of 58)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021)
- [Improved Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2102.09672/) (2021)
- [Fourier Neural Operator for Parametric Partial Differential Equations](https://mlsanity.com/paper/2010.08895/) (2020)
- [Exploring Simple Siamese Representation Learning](https://mlsanity.com/paper/2011.10566/) (2020)
- [Unsupervised Learning of Visual Features by Contrasting Cluster Assignments](https://mlsanity.com/paper/2006.09882/) (2020)
- [Improved Baselines with Momentum Contrastive Learning](https://mlsanity.com/paper/2003.04297/) (2020)
- [Barlow Twins: Self-Supervised Learning via Redundancy Reduction](https://mlsanity.com/paper/2103.03230/) (2021)
- [An Empirical Study of Training Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.02057/) (2021)
- [Graph Neural Networks: A Review of Methods and Applications](https://mlsanity.com/paper/1812.08434/) (2018)
- [Deep Sets](https://mlsanity.com/paper/1703.06114/) (2017)
- [Geometric Deep Learning: Grids, Groups, Graphs, Geodesics, and Gauges](https://mlsanity.com/paper/2104.13478/) (2021)
- [Group Equivariant Convolutional Networks](https://mlsanity.com/paper/1602.07576/) (2016)
- [VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learning](https://mlsanity.com/paper/2105.04906/) (2021)
- [E(n) Equivariant Graph Neural Networks](https://mlsanity.com/paper/2102.09844/) (2021)
- [Tensor field networks: Rotation- and translation-equivariant neural networks for 3D point clouds](https://mlsanity.com/paper/1802.08219/) (2018)
- [Large Batch Training of Convolutional Networks](https://mlsanity.com/paper/1708.03888/) (2017)
- [Deep learning for universal linear embeddings of nonlinear dynamics](https://mlsanity.com/paper/1712.09707/) (2017)
- [PDEBENCH: An Extensive Benchmark for Scientific Machine Learning](https://mlsanity.com/paper/2210.07182/) (2022)
- [ClimaX: A foundation model for weather and climate](https://mlsanity.com/paper/2301.10343/) (2023)
- [Provable Guarantees for Self-Supervised Deep Learning with Spectral Contrastive Loss](https://mlsanity.com/paper/2106.04156/) (2021)
- [PDE-Net: Learning PDEs from Data](https://mlsanity.com/paper/1710.09668/) (2017)
- [On the Generalization of Equivariance and Convolution in Neural Networks to the Action of Compact Groups](https://mlsanity.com/paper/1802.03690/) (2018)

## Built on by (most cited first, 29 of 29)

- [DPOT: Auto-Regressive Denoising Operator Transformer for Large-Scale PDE Pre-Training](https://mlsanity.com/paper/2403.03542/) (2024)
- [Multiple Physics Pretraining for Physical Surrogate Models](https://mlsanity.com/paper/2310.02994/) (2023)
- [Pretraining Codomain Attention Neural Operators for Solving Multiphysics PDEs](https://mlsanity.com/paper/2403.12553/) (2024)
- [Data-Efficient Operator Learning via Unsupervised Pretraining and In-Context Learning](https://mlsanity.com/paper/2402.15734/) (2024)
- [Lie Point Symmetry and Physics Informed Networks](https://mlsanity.com/paper/2311.04293/) (2023)
- [AI Foundation Models for Weather and Climate: Applications, Design, and Implementation](https://mlsanity.com/paper/2309.10808/) (2023)
- [DISCO: learning to DISCover an evolution Operator for multi-physics-agnostic prediction](https://mlsanity.com/paper/2504.19496/) (2025)
- [Learning Infinitesimal Generators of Continuous Symmetries from Data](https://mlsanity.com/paper/2410.21853/) (2024)
- [Symmetry-Informed Governing Equation Discovery](https://mlsanity.com/paper/2405.16756/) (2024)
- [Masked Autoencoders are PDE Learners](https://mlsanity.com/paper/2403.17728/) (2024)
- [Strategies for Pretraining Neural Operators](https://mlsanity.com/paper/2406.08473/) (2024)
- [Representation Learning for Spatiotemporal Physical Systems](https://mlsanity.com/paper/2603.13227/) (2026)
- [Test-time Generalization for Physics through Neural Operator Splitting](https://mlsanity.com/paper/2602.00884/) (2026)
- [OmniArch: Building Foundation Model For Scientific Computing](https://mlsanity.com/paper/2402.16014/) (2024)
- [Unitary convolutions for learning on graphs and groups](https://mlsanity.com/paper/2410.05499/) (2024)
- [Operator Learning with Domain Decomposition for Geometry Generalization in PDE Solving](https://mlsanity.com/paper/2504.00510/) (2025)
- [Discovering Symbolic Differential Equations with Symmetry Invariants](https://mlsanity.com/paper/2505.12083/) (2025)
- [Pretraining a Neural Operator in Lower Dimensions](https://mlsanity.com/paper/2407.17616/) (2024)
- [Physics-informed Temporal Alignment for Auto-regressive PDE Foundation Models](https://mlsanity.com/paper/2505.10930/) (2025)
- [A comparison of Single- and Double-generator formalisms for Thermodynamics-Informed Neural Networks](https://mlsanity.com/paper/2404.01060/) (2024)
- [Data-Efficient Inference of Neural Fluid Fields via SciML Foundation Model](https://mlsanity.com/paper/2412.13897/) (2024)
- [Smooth and Sparse Latent Dynamics in Operator Learning with Jerk Regularization](https://mlsanity.com/paper/2402.15636/) (2024)
- [Learning Data-Efficient and Generalizable Neural Operators via Fundamental Physics Knowledge](https://mlsanity.com/paper/2602.15184/) (2026)
- [Optimizing Likelihood-free Inference using Self-supervised Neural Symmetry Embeddings](https://mlsanity.com/paper/2312.07615/) (2023)
- [Scale-Consistent Learning for Partial Differential Equations](https://mlsanity.com/paper/2507.18813/) (2025)
- [AdamFLIP: Adaptive Momentum Feedback Linearization Optimization for Hard Constrained PINN Training](https://mlsanity.com/paper/2605.08408/) (2026)
- [Attention-enhanced neural differential equations for physics-informed deep learning of ion transport](https://mlsanity.com/paper/2312.02871/) (2023)
- [LieSolver: PDE-Constrained Learning for IBVPs via Lie Symmetries](https://mlsanity.com/paper/2510.25731/) (2025)
- [UniFluids: Unified Neural Operator Learning with Conditional Flow-matching](https://mlsanity.com/paper/2603.22309/) (2026)

## Abstract (arXiv)

Machine learning for differential equations paves the way for computationally efficient alternatives to numerical solvers, with potentially broad impacts in science and engineering. Though current algorithms typically require simulated training data tailored to a given setting, one may instead wish to learn useful information from heterogeneous sources, or from real dynamical systems observations that are messy or incomplete. In this work, we learn general-purpose representations of PDEs from heterogeneous data by implementing joint embedding methods for self-supervised learning (SSL), a framework for unsupervised representation learning that has had notable success in computer vision. Our representation outperforms baseline approaches to invariant tasks, such as regressing the coefficients of a PDE, while also improving the time-stepping performance of neural solvers. We hope that our proposed methodology will prove useful in the eventual development of general-purpose foundation models for PDEs. Code: https://github.com/facebookresearch/SSLForPDEs.