mlsanityData to 14 Oct 2026

Controlgym: Large-Scale Control Environments for Benchmarking Reinforcement Learning Algorithms

arXiv 2311.18736 ↗First posted 30 Nov 2023eess.SY cs.AI cs.CEcited by 6

Not on the map. Why?

Abstract (arXiv)

We introduce controlgym, a library of thirty-six industrial control settings, and ten infinite-dimensional partial differential equation (PDE)-based control problems. Integrated within the OpenAI Gym/Gymnasium (Gym) framework, controlgym allows direct applications of standard reinforcement learning (RL) algorithms like stable-baselines3. Our control environments complement those in Gym with continuous, unbounded action and observation spaces, motivated by real-world control applications. Moreover, the PDE control environments uniquely allow the users to extend the state dimensionality of the system to infinity while preserving the intrinsic dynamics. This feature is crucial for evaluating the scalability of RL algorithms for control. This project serves the learning for dynamics & control (L4DC) community, aiming to explore key questions: the convergence of RL algorithms in learning control policies; the stability and robustness issues of learning-based controllers; and the scalability of RL algorithms to high- and potentially infinite-dimensional systems. We open-source the controlgym project at https://github.com/xiangyuan-zhang/controlgym.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Builds on 48 of its 82 references that are in the map. First posted on arXiv 30 Nov 2023.

Builds on

48 of 82 references.

References · 48

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2015Trust Region Policy Optimizationcited by 2,580
  3. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,258
  4. 2020dm_control: Software and Tasks for Continuous Controlcited by 375
  5. 2016Benchmarking Deep Reinforcement Learning for Continuous Controlcited by 371
  6. 2021Safe Learning in Robotics: From Learning-Based Control to Safe Reinforcement Learningcited by 371
  7. 2018Global Convergence of Policy Gradient Methods for the Linear Quadratic Regulatorcited by 250
  8. 2017On the Sample Complexity of the Linear Quadratic Regulatorcited by 159
  9. 2018A Tour of Reinforcement Learning: The View from Continuous Controlcited by 151
  10. 2021Tianshou: a Highly Modularized Deep Reinforcement Learning Librarycited by 143
  11. 2020Naive Exploration is Optimal for Online LQRcited by 116
  12. 2019Online Control with Adversarial Disturbancescited by 92
36 more
  1. 2020Improper Learning for Non-Stochastic Controlcited by 83
  2. 2020An empirical investigation of the challenges of real-world reinforcement learningcited by 77
  3. 2018Derivative-Free Methods for Policy Optimization: Guarantees for Linear Quadratic Systemscited by 67
  4. 2023TorchRL: A data-driven decision-making library for PyTorchcited by 60
  5. 2022Statistical Learning Theory for Control: A Finite Sample Perspectivecited by 53
  6. 2020Policy Gradient Methods for the Noisy Linear Quadratic Regulator over a Finite Horizoncited by 50
  7. 2019Policy Optimization for H2 Linear Control with H_∞ Robustness Guarantee: Implicit Regularization and Global Convergencecited by 46
  8. 2019Convergence and sample complexity of gradient methods for the model-free linear quadratic regulator problemcited by 46
  9. 2020Black-Box Control for Linear Dynamical Systemscited by 43
  10. 2019Policy Optimization Provably Converges to Nash Equilibria in Zero-Sum Linear Quadratic Gamescited by 42
  11. 2018The Gap Between Model-Based and Model-Free Methods on the Linear Quadratic Regulator: An Asymptotic Viewpointcited by 40
  12. 2019LQR through the Lens of First Order Methods: Discrete-time Casecited by 40
  13. 2019Distributed Reinforcement Learning for Decentralized Linear Quadratic Control: A Derivative-Free Policy Optimization Approachcited by 38
  14. 2019Control of chaotic systems by Deep Reinforcement Learningcited by 29
  15. 2021Analysis of the Optimization Landscape of Linear Quadratic Gaussian (LQG) Controlcited by 28
  16. 2021Stabilizing Dynamical Systems via Policy Gradient Methodscited by 25
  17. 2023Distributed Control of Partial Differential Equations Using Convolutional Reinforcement Learningcited by 24
  18. 2021Physics-informed Dyna-Style Model-Based Deep Reinforcement Learning for Dynamic Controlcited by 23
  19. 2019Learning the Globally Optimal Distributed LQ Regulatorcited by 20
  20. 2020Reinforcement Learning with Fast Stabilization in Linear Dynamical Systemscited by 20
  21. 2022Data-driven control of spatiotemporal chaos with reduced-order neural ODE-based models and reinforcement learningcited by 17
  22. 2019Global Convergence of Policy Gradient for Sequential Zero-Sum Linear Quadratic Dynamic Gamescited by 16
  23. 2018Reinforcement Learning with Function-Valued Action Spaces for Partial Differential Equation Controlcited by 11
  24. 2022On the Optimization Landscape of Dynamic Output Feedback Linear Quadratic Controlcited by 11
  25. 2021Deluca -- A Differentiable Control Library: Environments, Methods, and Benchmarkingcited by 10
  26. 2022How are policy gradient methods affected by the limits of control?cited by 10
  27. 2023Revisiting LQR Control from the Perspective of Receding-Horizon Policy Gradientcited by 10
  28. 2021Optimization Landscape of Gradient Descent for Discrete-time Static Output Feedbackcited by 9
  29. 2023Learning the Kalman Filter with Fine-Grained Sample Complexitycited by 9
  30. 2020Primal-dual Learning for the Model-free Risk-constrained Linear Quadratic Regulatorcited by 8
  31. 2022Globally Convergent Policy Search over Dynamic Filters for Output Estimationcited by 8
  32. 2022Global Convergence of Direct Policy Search for State-Feedback H_∞ Robust Control: A Revisit of Nonsmooth Synthesis with Goldstein Subdifferentialcited by 7
  33. 2023Learning a model is paramount for sample efficiency in reinforcement learning control of PDEscited by 6
  34. 2021Model-Free μ Synthesis via Adversarial Reinforcement Learningcited by 5
  35. 2023Global Convergence of Receding-Horizon Policy Search in Learning Estimator Designscited by 4
  36. 2023Reinforcement learning-based estimation for partial differential equationscited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 4.

6 papers.

  1. 2025Time-Aware World Model for Adaptive Prediction and Controlcited by 4
  2. 2026Plug-and-Play Benchmarking of Reinforcement Learning Algorithms for Large-Scale Flow Controlcited by 2
  3. 2026Vintix II: Decision Pre-Trained Transformer is a Scalable In-Context Reinforcement Learnercited by 1
  4. 2025ASPEN: An Adaptive Spectral Physics-Enabled Network for Ginzburg-Landau Dynamicscited by 0
  5. 2026Two-Layer Linear Auto-Regressive Models Estimate Latent Statescited by 0
  6. 2026Backstepping-Guided Reinforcement Learning for Wide-Range Saint-Venant Canal Regulationcited by 0