# Physics-informed attention-based neural network for solving non-linear partial differential equations

arXiv: https://arxiv.org/abs/2105.07898 · HTML: https://mlsanity.com/paper/2105.07898/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 35 references that are in the map.
- First posted on arXiv 17 May 2021.

## Builds on (most cited first, 19 of 19)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [Sequence to Sequence Learning with Neural Networks](https://mlsanity.com/paper/1409.3215/) (2014)
- [Solving high-dimensional partial differential equations using deep learning](https://mlsanity.com/paper/1707.02568/) (2017)
- [A Survey of Deep Learning Techniques for Autonomous Driving](https://mlsanity.com/paper/1910.07738/) (2019)
- [Machine Learning for Fluid Mechanics](https://mlsanity.com/paper/1905.11075/) (2019)
- [Physics Informed Deep Learning (Part I): Data-driven Solutions of Nonlinear Partial Differential Equations](https://mlsanity.com/paper/1711.10561/) (2017)
- [B-PINNs: Bayesian Physics-Informed Neural Networks for Forward and Inverse PDE Problems with Noisy Data](https://mlsanity.com/paper/2003.06097/) (2020)
- [Quantifying total uncertainty in physics-informed neural networks for solving forward and inverse stochastic problems](https://mlsanity.com/paper/1809.08327/) (2018)
- [An Energy Approach to the Solution of Partial Differential Equations in Computational Mechanics via Machine Learning: Concepts, Implementation and Applications](https://mlsanity.com/paper/1908.10407/) (2019)
- [An overview on deep learning-based approximation methods for partial differential equations](https://mlsanity.com/paper/2012.12348/) (2020)
- [The Unreasonable Effectiveness of Deep Learning in Artificial Intelligence](https://mlsanity.com/paper/2002.04806/) (2020)
- [Bootstrapping Conditional GANs for Video Game Level Generation](https://mlsanity.com/paper/1910.01603/) (2019)
- [Deep Reinforcement Learning for General Video Game AI](https://mlsanity.com/paper/1806.02448/) (2018)
- [Physics-informed Neural Networks for Solving Nonlinear Diffusivity and Biot's equations](https://mlsanity.com/paper/2002.08235/) (2020)
- [Physics Informed Deep Learning for Transport in Porous Media. Buckley Leverett Problem](https://mlsanity.com/paper/2001.05172/) (2020)
- [Conditional Karhunen-Lo`eve expansion for uncertainty quantification and active learning in partial differential equation models](https://mlsanity.com/paper/1904.08069/) (2019)

## Built on by (most cited first, 6 of 6)

- [Physics-Informed Neural Networks with Adaptive Localized Artificial Viscosity](https://mlsanity.com/paper/2203.08802/) (2022)
- [Physical Activation Functions (PAFs): An Approach for More Efficient Induction of Physics into Physics-Informed Neural Networks (PINNs)](https://mlsanity.com/paper/2205.14630/) (2022)
- [GrADE: A graph based data-driven solver for time-dependent nonlinear partial differential equations](https://mlsanity.com/paper/2108.10639/) (2021)
- [Robust Biharmonic Skinning Using Geometric Fields](https://mlsanity.com/paper/2406.00238/) (2024)
- [Discovering Physics-Informed Neural Networks Model for Solving Partial Differential Equations through Evolutionary Computation](https://mlsanity.com/paper/2405.11208/) (2024)
- [A Critical Review of Physics-Informed Machine Learning Applications in Subsurface Energy Systems](https://mlsanity.com/paper/2308.04457/) (2023)

## Abstract (arXiv)

Physics-Informed Neural Networks (PINNs) have enabled significant improvements in modelling physical processes described by partial differential equations (PDEs). PINNs are based on simple architectures, and learn the behavior of complex physical systems by optimizing the network parameters to minimize the residual of the underlying PDE. Current network architectures share some of the limitations of classical numerical discretization schemes when applied to non-linear differential equations in continuum mechanics. A paradigmatic example is the solution of hyperbolic conservation laws that develop highly localized nonlinear shock waves. Learning solutions of PDEs with dominant hyperbolic character is a challenge for current PINN approaches, which rely, like most grid-based numerical schemes, on adding artificial dissipation. Here, we address the fundamental question of which network architectures are best suited to learn the complex behavior of non-linear PDEs. We focus on network architecture rather than on residual regularization. Our new methodology, called Physics-Informed Attention-based Neural Networks, (PIANNs), is a combination of recurrent neural networks and attention mechanisms. The attention mechanism adapts the behavior of the deep neural network to the non-linear features of the solution, and break the current limitations of PINNs. We find that PIANNs effectively capture the shock front in a hyperbolic model problem, and are capable of providing high-quality solutions inside and beyond the training set.