# Neural Operator based Reinforcement Learning for Control of first-order PDEs with Spatially-Varying State Delay

arXiv: https://arxiv.org/abs/2501.18201 · HTML: https://mlsanity.com/paper/2501.18201/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods.
- Builds on 8 of its 14 references that are in the map.
- First posted on arXiv 30 Jan 2025.

## Placed in

- Deep reinforcement learning methods / [RL theory and sample complexity](https://mlsanity.com/deeprl/rl-theory-and-sample-complexity/)

## Builds on (most cited first, 8 of 8)

- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Reaching the Limit in Autonomous Racing: Optimal Control versus Reinforcement Learning](https://mlsanity.com/paper/2310.10943/) (2023)
- [Neural Operators for Bypassing Gain and Control Computations in PDE Backstepping](https://mlsanity.com/paper/2302.14265/) (2023)
- [Deep Reinforcement Learning with Shallow Controllers: An Experimental Application to PID Tuning](https://mlsanity.com/paper/2111.07171/) (2021)
- [Distributed Control of Partial Differential Equations Using Convolutional Reinforcement Learning](https://mlsanity.com/paper/2301.10737/) (2023)
- [Reinforcement Learning with Fast Stabilization in Linear Dynamical Systems](https://mlsanity.com/paper/2007.12291/) (2020)
- [Neural Operators for PDE Backstepping Control of First-Order Hyperbolic PIDE with Recycle and Delay](https://mlsanity.com/paper/2307.11436/) (2023)
- [Proximal Policy Optimization Learning based Control of Congested Freeway Traffic](https://mlsanity.com/paper/2204.05627/) (2022)

## Abstract (arXiv)

Control of distributed parameter systems affected by delays is a challenging task, particularly when the delays depend on spatial variables. The idea of integrating analytical control theory with learning-based control within a unified control scheme is becoming increasingly promising and advantageous. In this paper, we address the problem of controlling an unstable first-order hyperbolic PDE with spatially-varying delays by combining PDE backstepping control strategies and deep reinforcement learning (RL). To eliminate the assumption on the delay function required for the backstepping design, we propose a soft actor-critic (SAC) architecture incorporating a DeepONet to approximate the backstepping controller. The DeepONet extracts features from the backstepping controller and feeds them into the policy network. In simulations, our algorithm outperforms the baseline SAC without prior backstepping knowledge and the analytical controller.