mlsanityData to 14 Oct 2026

Self-Supervised Learning with Lie Symmetries for Partial Differential Equations

arXiv 2307.05432 ↗First posted 11 Jul 2023cs.LG cs.NA math.NAcited by 29

Not on the map. Why?

Abstract (arXiv)

Machine learning for differential equations paves the way for computationally efficient alternatives to numerical solvers, with potentially broad impacts in science and engineering. Though current algorithms typically require simulated training data tailored to a given setting, one may instead wish to learn useful information from heterogeneous sources, or from real dynamical systems observations that are messy or incomplete. In this work, we learn general-purpose representations of PDEs from heterogeneous data by implementing joint embedding methods for self-supervised learning (SSL), a framework for unsupervised representation learning that has had notable success in computer vision. Our representation outperforms baseline approaches to invariant tasks, such as regressing the coefficients of a PDE, while also improving the time-stepping performance of neural solvers. We hope that our proposed methodology will prove useful in the eventual development of general-purpose foundation models for PDEs. Code: https://github.com/facebookresearch/SSLForPDEs.

Key facts. Cited by 29 papers in the map (data through 14 Oct 2026). Builds on 58 of its 105 references that are in the map. First posted on arXiv 11 Jul 2023.

Builds on

58 of 105 references.

References · 58

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  4. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,974
  5. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  6. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  7. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  8. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,750
  9. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
  10. 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,761
  11. 2020Fourier Neural Operator for Parametric Partial Differential Equationscited by 2,908
  12. 2020Exploring Simple Siamese Representation Learningcited by 2,866
38 more
  1. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,857
  2. 2020Improved Baselines with Momentum Contrastive Learningcited by 2,287
  3. 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,838
  4. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,616
  5. 2018Graph Neural Networks: A Review of Methods and Applicationscited by 1,576
  6. 2017Deep Setscited by 1,421
  7. 2021Geometric Deep Learning: Grids, Groups, Graphs, Geodesics, and Gaugescited by 1,167
  8. 2016Group Equivariant Convolutional Networkscited by 1,165
  9. 2021VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learningcited by 1,057
  10. 2021E(n) Equivariant Graph Neural Networkscited by 965
  11. 2018Tensor field networks: Rotation- and translation-equivariant neural networks for 3D point cloudscited by 746
  12. 2017Large Batch Training of Convolutional Networkscited by 535
  13. 2017Deep learning for universal linear embeddings of nonlinear dynamicscited by 494
  14. 2022PDEBENCH: An Extensive Benchmark for Scientific Machine Learningcited by 413
  15. 2023ClimaX: A foundation model for weather and climatecited by 337
  16. 2021Provable Guarantees for Self-Supervised Deep Learning with Spectral Contrastive Losscited by 313
  17. 2017PDE-Net: Learning PDEs from Datacited by 311
  18. 2018On the Generalization of Equivariance and Convolution in Neural Networks to the Action of Compact Groupscited by 300
  19. 2015Unitary Evolution Recurrent Neural Networkscited by 294
  20. 2020Whitening for Self-Supervised Representation Learningcited by 279
  21. 2023A Cookbook of Self-Supervised Learningcited by 250
  22. 2022Towards Multi-spatiotemporal-scale Generalized PDE Modelingcited by 197
  23. 2019Symplectic Recurrent Neural Networkscited by 150
  24. 2021Decoupled Contrastive Learningcited by 143
  25. 2021Enhancing Computational Fluid Dynamics with Machine Learningcited by 138
  26. 2020Incorporating Symmetry into Deep Dynamics Models for Improved Generalizationcited by 126
  27. 2018Deep Fluids: A Generative Network for Parameterized Fluid Simulationscited by 121
  28. 2022VICRegL: Self-Supervised Learning of Local Visual Featurescited by 118
  29. 2022On the duality between contrastive and non-contrastive self-supervised learningcited by 108
  30. 2020Simplifying Hamiltonian and Lagrangian Neural Networks via Explicit Constraintscited by 105
  31. 2020Efficient Riemannian Optimization on the Stiefel Manifold via the Cayley Transformcited by 80
  32. 2022Lie Point Symmetry Data Augmentation for Neural PDE Solverscited by 77
  33. 2022Neural Conservation Laws: A Divergence-Free Perspectivecited by 70
  34. 2020A survey on Semi-, Self- and Unsupervised Learning for Image Classificationcited by 55
  35. 2022Neural Manifold Clustering and Embeddingcited by 44
  36. 2023Self-supervised learning of Split Invariant Equivariant representationscited by 44
  37. 2018Variational Autoencoders for New Physics Mining at the Large Hadron Collidercited by 41
  38. 2023The SSL Interplay: Augmentations, Inductive Bias, and Generalizationcited by 40

50 most cited of 58 All 58 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 5; 2024: 13; 2025: 6; 2026: 5.

29 papers.

Sort:

By area:

  1. 2024DPOT: Auto-Regressive Denoising Operator Transformer for Large-Scale PDE Pre-Trainingcited by 151
  2. 2023Multiple Physics Pretraining for Physical Surrogate Modelscited by 98
  3. 2024Pretraining Codomain Attention Neural Operators for Solving Multiphysics PDEscited by 59
  4. 2024Data-Efficient Operator Learning via Unsupervised Pretraining and In-Context Learningcited by 32
  5. 2023Lie Point Symmetry and Physics Informed Networkscited by 24
  6. 2023AI Foundation Models for Weather and Climate: Applications, Design, and Implementationcited by 21
  7. 2025DISCO: learning to DISCover an evolution Operator for multi-physics-agnostic predictioncited by 21
  8. 2024Learning Infinitesimal Generators of Continuous Symmetries from Datacited by 17
  9. 2024Symmetry-Informed Governing Equation Discoverycited by 15
  10. 2024Masked Autoencoders are PDE Learnerscited by 14
  11. 2024Strategies for Pretraining Neural Operatorscited by 14
  12. 2026Representation Learning for Spatiotemporal Physical Systemscited by 11
17 more
  1. 2026Test-time Generalization for Physics through Neural Operator Splittingcited by 10
  2. 2024OmniArch: Building Foundation Model For Scientific Computingcited by 9
  3. 2024Unitary convolutions for learning on graphs and groupscited by 8
  4. 2025Operator Learning with Domain Decomposition for Geometry Generalization in PDE Solvingcited by 6
  5. 2025Discovering Symbolic Differential Equations with Symmetry Invariantscited by 6
  6. 2024Pretraining a Neural Operator in Lower Dimensionscited by 5
  7. 2025Physics-informed Temporal Alignment for Auto-regressive PDE Foundation Modelscited by 5
  8. 2024A comparison of Single- and Double-generator formalisms for Thermodynamics-Informed Neural Networkscited by 4
  9. 2024Data-Efficient Inference of Neural Fluid Fields via SciML Foundation Modelcited by 4
  10. 2024Smooth and Sparse Latent Dynamics in Operator Learning with Jerk Regularizationcited by 3
  11. 2026Learning Data-Efficient and Generalizable Neural Operators via Fundamental Physics Knowledgecited by 2
  12. 2023Optimizing Likelihood-free Inference using Self-supervised Neural Symmetry Embeddingscited by 1
  13. 2025Scale-Consistent Learning for Partial Differential Equationscited by 1
  14. 2026AdamFLIP: Adaptive Momentum Feedback Linearization Optimization for Hard Constrained PINN Trainingcited by 1
  15. 2023Attention-enhanced neural differential equations for physics-informed deep learning of ion transportcited by 0
  16. 2025LieSolver: PDE-Constrained Learning for IBVPs via Lie Symmetriescited by 0
  17. 2026UniFluids: Unified Neural Operator Learning with Conditional Flow-matchingcited by 0