# Backpropagation through Time and Space: Learning Numerical Methods with Multi-Agent Reinforcement Learning

arXiv: https://arxiv.org/abs/2203.08937 · HTML: https://mlsanity.com/paper/2203.08937/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 30 references that are in the map.
- First posted on arXiv 16 Mar 2022.

## Builds on (most cited first, 13 of 13)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Fourier Neural Operator for Parametric Partial Differential Equations](https://mlsanity.com/paper/2010.08895/) (2020)
- [Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments](https://mlsanity.com/paper/1706.02275/) (2017)
- [Counterfactual Multi-Agent Policy Gradients](https://mlsanity.com/paper/1705.08926/) (2017)
- [QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/1803.11485/) (2018)
- [Learning to Communicate with Deep Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/1605.06676/) (2016)
- [Deep Recurrent Q-Learning for Partially Observable MDPs](https://mlsanity.com/paper/1507.06527/) (2015)
- [Emergence of Grounded Compositional Language in Multi-Agent Populations](https://mlsanity.com/paper/1703.04908/) (2017)
- [Physics Informed Deep Learning (Part II): Data-driven Discovery of Nonlinear Partial Differential Equations](https://mlsanity.com/paper/1711.10566/) (2017)
- [Scientific multi-agent reinforcement learning for wall-models of turbulent flows](https://mlsanity.com/paper/2106.11144/) (2021)
- [Remember and Forget for Experience Replay](https://mlsanity.com/paper/1807.05827/) (2018)
- [Learning to Discretize: Solving 1D Scalar Conservation Laws via Deep Reinforcement Learning](https://mlsanity.com/paper/1905.11079/) (2019)
- [Solving Irregular and Data-enriched Differential Equations using Deep Neural Networks](https://mlsanity.com/paper/1905.04351/) (2019)

## Built on by (most cited first, 1 of 1)

- [Multi-Agent Learning of Numerical Methods for Hyperbolic PDEs with Factored Dec-MDP](https://mlsanity.com/paper/2205.15716/) (2022)

## Abstract (arXiv)

We introduce Backpropagation Through Time and Space (BPTTS), a method for training a recurrent spatio-temporal neural network, that is used in a homogeneous multi-agent reinforcement learning (MARL) setting to learn numerical methods for hyperbolic conservation laws. We treat the numerical schemes underlying partial differential equations (PDEs) as a Partially Observable Markov Game (POMG) in Reinforcement Learning (RL). Similar to numerical solvers, our agent acts at each discrete location of a computational space for efficient and generalizable learning. To learn higher-order spatial methods by acting on local states, the agent must discern how its actions at a given spatiotemporal location affect the future evolution of the state. The manifestation of this non-stationarity is addressed by BPTTS, which allows for the flow of gradients across both space and time. The learned numerical policies are comparable to the SOTA numerics in two settings, the Burgers' Equation and the Euler Equations, and generalize well to other simulation set-ups.