mlsanityData to 7 Oct 2026

Learning a model is paramount for sample efficiency in reinforcement learning control of PDEs

arXiv 2302.07160 ↗First posted 14 Feb 2023cs.LGcited by 5

Not on the map. Why?

Abstract (arXiv)

The goal of this paper is to make a strong point for the usage of dynamical models when using reinforcement learning (RL) for feedback control of dynamical systems governed by partial differential equations (PDEs). To breach the gap between the immense promises we see in RL and the applicability in complex engineering systems, the main challenges are the massive requirements in terms of the training data, as well as the lack of performance guarantees. We present a solution for the first issue using a data-driven surrogate model in the form of a convolutional LSTM with actuation. We demonstrate that learning an actuated model in parallel to training the RL agent significantly reduces the total amount of required data sampled from the real system. Furthermore, we show that iteratively updating the model is of major importance to avoid biases in the RL training. Detailed ablation studies reveal the most important ingredients of the modeling process. We use the chaotic Kuramoto-Sivashinsky equation do demonstarte our findings.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 45 of its 79 references that are in the map. First posted on arXiv 14 Feb 2023.

Builds on

45 of 79 references.

References · 45

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2017Proximal Policy Optimization Algorithmscited by 14,441
  4. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  5. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
  6. 2016Layer Normalizationcited by 5,579
  7. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  8. 2015Continuous control with deep reinforcement learningcited by 3,279
  9. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  10. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,704
  11. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,598
  12. 2016Neural Architecture Search with Reinforcement Learningcited by 1,489
33 more
  1. 2015Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcastingcited by 1,428
  2. 2018Learning Latent Dynamics for Planning from Pixelscited by 1,221
  3. 2019Dota 2 with Large Scale Deep Reinforcement Learningcited by 1,084
  4. 2017Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learningcited by 935
  5. 2019When to Trust Your Model: Model-Based Policy Optimizationcited by 753
  6. 2018Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Modelscited by 748
  7. 2020NVAE: A Deep Hierarchical Variational Autoencodercited by 635
  8. 2016Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networkscited by 541
  9. 2019Model-Based Reinforcement Learning for Ataricited by 495
  10. 2015Learning Deconvolution Network for Semantic Segmentationcited by 365
  11. 2017Neural Network Dynamics for Model-Based Deep Reinforcement Learning with Model-Free Fine-Tuningcited by 288
  12. 2018Model-Ensemble Trust-Region Policy Optimizationcited by 214
  13. 2019Benchmarking Model-Based Reinforcement Learningcited by 158
  14. 2017Sparse identification of nonlinear dynamics for model predictive control in the low-data limitcited by 139
  15. 2016Linear predictors for nonlinear dynamical systems: Koopman operator meets model predictive controlcited by 134
  16. 2014Dynamic mode decomposition with controlcited by 128
  17. 2019Modeling the Dynamics of PDE Systems with Physics-Constrained Deep Auto-Regressive Networkscited by 127
  18. 2019When to use parametric models in reinforcement learning?cited by 118
  19. 2021PhyCRNet: Physics-informed Convolutional-Recurrent Network for Solving Spatiotemporal PDEscited by 97
  20. 2019Exploring Model-based Planning with Policy Networkscited by 90
  21. 2017Recurrent Environment Simulatorscited by 68
  22. 2019A review on Deep Reinforcement Learning for Fluid Mechanicscited by 65
  23. 2017Koopman operator-based model reduction for switched-system control of PDEscited by 31
  24. 2019Control of chaotic systems by Deep Reinforcement Learningcited by 28
  25. 2018Deep autoregressive neural networks for high-dimensional inverse problems in groundwater contaminant source identificationcited by 26
  26. 2021Finite-data error bounds for Koopman-based prediction and controlcited by 26
  27. 2021Physics-informed Dyna-Style Model-Based Deep Reinforcement Learning for Dynamic Controlcited by 22
  28. 2023Distributed Control of Partial Differential Equations Using Convolutional Reinforcement Learningcited by 21
  29. 2022Data-driven control of spatiotemporal chaos with reduced-order neural ODE-based models and reinforcement learningcited by 16
  30. 2018The Effect of Planning Shape on Dyna-style Planning in High-dimensional State Spacescited by 15
  31. 2021Symmetry reduction for deep reinforcement learning active control of chaotic spatiotemporal dynamicscited by 10
  32. 2022Direct data-driven forecast of local turbulent heat flux in Rayleigh-B'enard convectioncited by 10
  33. 2021On the Universal Transformation of Data-Driven Models to Control Systemscited by 5

Built on by

Cited by year

Citing papers per year: 2023: 2; 2024: 2; 2025: 1.

5 papers.

  1. 2023Controlgym: Large-Scale Control Environments for Benchmarking Reinforcement Learning Algorithmscited by 4
  2. 2024Solving Partial Differential Equations with Equivariant Extreme Learning Machinescited by 2
  3. 2024Interpretable and Efficient Data-driven Discovery and Control of Distributed Systemscited by 2
  4. 2023Taming Waves: A Physically-Interpretable Machine Learning Framework for Realizable Control of Wave Dynamicscited by 0
  5. 2025Acoustic Wave Manipulation Through Sparse Robotic Actuationcited by 0