# Distributional Offline Continuous-Time Reinforcement Learning with Neural Physics-Informed PDEs (SciPhy RL for DOCTR-L)

arXiv: https://arxiv.org/abs/2104.01040 · HTML: https://mlsanity.com/paper/2104.01040/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 36 references that are in the map.
- First posted on arXiv 2 Apr 2021.

## Builds on (most cited first, 16 of 16)

- [Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems](https://mlsanity.com/paper/2005.01643/) (2020)
- [Off-Policy Deep Reinforcement Learning without Exploration](https://mlsanity.com/paper/1812.02900/) (2018)
- [A Distributional Perspective on Reinforcement Learning](https://mlsanity.com/paper/1707.06887/) (2017)
- [DeepXDE: A deep learning library for solving differential equations](https://mlsanity.com/paper/1907.04502/) (2019)
- [Solving high-dimensional partial differential equations using deep learning](https://mlsanity.com/paper/1707.02568/) (2017)
- [Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Review](https://mlsanity.com/paper/1805.00909/) (2018)
- [Keep Doing What Worked: Behavioral Modelling Priors for Offline Reinforcement Learning](https://mlsanity.com/paper/2002.08396/) (2020)
- [Deep learning-based numerical methods for high-dimensional parabolic partial differential equations and backward stochastic differential equations](https://mlsanity.com/paper/1706.04702/) (2017)
- [Why and When Can Deep -- but Not Shallow -- Networks Avoid the Curse of Dimensionality: a Review](https://mlsanity.com/paper/1611.00740/) (2016)
- [Risk-sensitive Reinforcement Learning](https://mlsanity.com/paper/1311.2097/) (2013)
- [Taming the Noise in Reinforcement Learning via Soft Updates](https://mlsanity.com/paper/1512.08562/) (2015)
- [Algorithms for Solving High Dimensional PDEs: From Nonlinear Monte Carlo to Machine Learning](https://mlsanity.com/paper/2008.13333/) (2020)
- [An overview on deep learning-based approximation methods for partial differential equations](https://mlsanity.com/paper/2012.12348/) (2020)
- [Risk-Averse Offline Reinforcement Learning](https://mlsanity.com/paper/2102.05371/) (2021)
- [Forward-Backward Stochastic Neural Networks: Deep Learning of High-dimensional Partial Differential Equations](https://mlsanity.com/paper/1804.07010/) (2018)
- [Hamilton-Jacobi-Bellman Equations for Maximum Entropy Optimal Control](https://mlsanity.com/paper/2009.13097/) (2020)

## Built on by (most cited first, 6 of 6)

- [Is L2 Physics-Informed Loss Always Suitable for Training Physics-Informed Neural Network?](https://mlsanity.com/paper/2206.02016/) (2022)
- [Distributional Hamilton-Jacobi-Bellman Equations for Continuous-Time Reinforcement Learning](https://mlsanity.com/paper/2205.12184/) (2022)
- [Action Gaps and Advantages in Continuous-Time Distributional Reinforcement Learning](https://mlsanity.com/paper/2410.11022/) (2024)
- [Tractable Representations for Convergent Approximation of Distributional HJB Equations](https://mlsanity.com/paper/2503.05563/) (2025)
- [RLOP: RL Methods in Option Pricing from a Mathematical Perspective](https://mlsanity.com/paper/2205.05600/) (2022)
- [A Meta Reinforcement Learning Approach to Goals-Based Wealth Management](https://mlsanity.com/paper/2605.02300/) (2026)

## Abstract (arXiv)

This paper addresses distributional offline continuous-time reinforcement learning (DOCTR-L) with stochastic policies for high-dimensional optimal control. A soft distributional version of the classical Hamilton-Jacobi-Bellman (HJB) equation is given by a semilinear partial differential equation (PDE). This `soft HJB equation' can be learned from offline data without assuming that the latter correspond to a previous optimal or near-optimal policy. A data-driven solution of the soft HJB equation uses methods of Neural PDEs and Physics-Informed Neural Networks developed in the field of Scientific Machine Learning (SciML). The suggested approach, dubbed `SciPhy RL', thus reduces DOCTR-L to solving neural PDEs from data. Our algorithm called Deep DOCTR-L converts offline high-dimensional data into an optimal policy in one step by reducing it to supervised learning, instead of relying on value iteration or policy iteration methods. The method enables a computable approach to the quality control of obtained policies in terms of both their expected returns and uncertainties about their values.