mlsanityData to 7 Oct 2026

Optimal Exploration for Model-Based RL in Nonlinear Systems

arXiv 2306.09210 ↗First posted 15 Jun 2023cs.LG cs.RO cs.SYcited by 17

Not on the map. Why?

Abstract (arXiv)

Learning to control unknown nonlinear dynamical systems is a fundamental problem in reinforcement learning and control theory. A commonly applied approach is to first explore the environment (exploration), learn an accurate model of it (system identification), and then compute an optimal controller with the minimum cost on this estimated system (policy optimization). While existing work has shown that it is possible to learn a uniformly good model of the system~citepmania2020active, in practice, if we aim to learn a good controller with a low cost on the actual system, certain system parameters may be significantly more critical than others, and we therefore ought to focus our exploration on learning such parameters. In this work, we consider the setting of nonlinear dynamical systems and seek to formally quantify, in such settings, (a) which parameters are most relevant to learning a good controller, and (b) how we can best explore so as to minimize uncertainty in such parameters. Inspired by recent work in linear systems~citepwagenmaker2021task, we show that minimizing the controller loss in nonlinear systems translates to estimating the system parameters in a particular, task-dependent metric. Motivated by this, we develop an algorithm able to efficiently explore the system to reduce uncertainty in this metric, and prove a lower bound showing that our approach learns a controller at a near-instance-optimal rate. Our algorithm relies on a general reduction from policy optimization to optimal experiment design in arbitrary systems, and may be of independent interest. We conclude with experiments demonstrating the effectiveness of our method in realistic nonlinear robotic systems.

Key facts. Cited by 17 papers in the map (data through 7 Oct 2026). Builds on 41 of its 69 references that are in the map. First posted on arXiv 15 Jun 2023.

Builds on

41 of 69 references.

References · 41

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2015Trust Region Policy Optimizationcited by 2,503
  3. 2018Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Modelscited by 748
  4. 2020MOPO: Model-based Offline Policy Optimizationcited by 630
  5. 2017Minimax Regret Bounds for Reinforcement Learningcited by 500
  6. 2019Model-Based Reinforcement Learning for Ataricited by 495
  7. 2021Safe Learning in Robotics: From Learning-Based Control to Safe Reinforcement Learningcited by 337
  8. 2018Provably Efficient Maximum Entropy Explorationcited by 240
  9. 2021The Statistical Complexity of Interactive Decision Makingcited by 208
  10. 2020Nearly Minimax Optimal Reinforcement Learning for Linear Mixture Markov Decision Processescited by 199
  11. 2019Tighter Problem-Dependent Regret Bounds in Reinforcement Learning without Domain Knowledge using Value Function Boundscited by 194
  12. 2021Bilinear Classes: A Structural Framework for Provable Generalization in RLcited by 176
29 more
  1. 2018Learning Without Mixing: Towards A Sharp Analysis of Linear System Identificationcited by 158
  2. 2017On the Sample Complexity of the Linear Quadratic Regulatorcited by 151
  3. 2019Model-Based Reinforcement Learning with a Generative Model is Minimax Optimalcited by 124
  4. 2014Model-based Reinforcement Learning and the Eluder Dimensioncited by 116
  5. 2020Naive Exploration is Optimal for Online LQRcited by 111
  6. 2020Information Theoretic Regret Bounds for Online Nonlinear Controlcited by 87
  7. 2020Improper Learning for Non-Stochastic Controlcited by 83
  8. 2018Neural Lander: Stable Drone Landing Control using Learned Dynamicscited by 82
  9. 2021Reward is enough for convex MDPscited by 82
  10. 2022Neural-Fly Enables Rapid Learning for Agile Flight in Strong Windscited by 80
  11. 2020Active Learning for Nonlinear System Identification with Guaranteescited by 78
  12. 2019Learning Linear-Quadratic Regulators Efficiently with only √T Regretcited by 67
  13. 2019Learning Linear Dynamical Systems with Semi-Parametric Least Squarescited by 58
  14. 2020Learning nonlinear dynamical systems from a single trajectorycited by 46
  15. 2022Leveraging Offline Data in Online Reinforcement Learningcited by 42
  16. 2020Non-asymptotic and Accurate Learning of Nonlinear Dynamical Systemscited by 39
  17. 2021Adaptive-Control-Oriented Meta-Learning for Nonlinear Systemscited by 38
  18. 2020Active Learning for Identification of Linear Dynamical Systemscited by 33
  19. 2022Instance-Dependent Near-Optimal Policy Identification in Linear MDPs via Online Experiment Designcited by 33
  20. 2020The Power of Predictions in Online Controlcited by 31
  21. 2022A General Framework for Sample-Efficient Function Approximation in Reinforcement Learningcited by 31
  22. 2020Regret Bounds for Adaptive Nonlinear Controlcited by 25
  23. 2021Meta-Adaptive Nonlinear Control: Theory and Algorithmscited by 23
  24. 2020Neural-Swarm2: Planning and Control of Heterogeneous Multirotor Swarms using Learned Interactionscited by 19
  25. 2018Stochastic Gradient Descent Learns State Equations with Nonlinear Activationscited by 16
  26. 2020Chance-Constrained Trajectory Optimization for Safe Exploration and Learning of Nonlinear Systemscited by 15
  27. 2021Task-Optimal Exploration in Linear Dynamical Systemscited by 15
  28. 2021PC-MLP: Model-based Reinforcement Learning with Policy Cover Guided Explorationcited by 14
  29. 2023On the Sample Complexity of the Linear Quadratic Gaussian Regulatorcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 10; 2025: 5; 2026: 1.

17 papers.

Sort:

By area:

  1. 2023Optimistic Active Exploration of Dynamical Systemscited by 32
  2. 2024MaxInfoRL: Boosting exploration in reinforcement learning through information gain maximizationcited by 30
  3. 2024ASID: Active Exploration for System Identification in Robotic Manipulationcited by 27
  4. 2024Overcoming the Sim-to-Real Gap: Leveraging Simulation to Learn to Explore for Real-World RLcited by 17
  5. 2024Distributional Successor Features Enable Zero-Shot Policy Optimizationcited by 8
  6. 2024Active Learning for Control-Oriented Identification of Nonlinear Systemscited by 7
  7. 2025SOMBRL: Scalable and Optimistic Model-Based RLcited by 7
  8. 2024NeoRL: Efficient Exploration for Nonepisodic RLcited by 3
  9. 2024First, Learn What You Don't Know: Active Information Gathering for Driving at the Limits of Handlingcited by 3
  10. 2024Guarantees for Nonlinear Representation Learning: Non-identical Covariates, Dependent Data, Fewer Samplescited by 1
  11. 2024Exploring the Edges of Latent State Clusters for Goal-Conditioned Reinforcement Learningcited by 1
  12. 2024Active learning of neural population dynamics using two-photon holographic optogeneticscited by 0
5 more
  1. 2025Digital Twin Calibration with Model-Based Reinforcement Learningcited by 0
  2. 2025Logarithmic Regret for Nonlinear Controlcited by 0
  3. 2025Model-Based Offline Reinforcement Learning with Reliability-Guaranteed Sequence Modelingcited by 0
  4. 2025PrefPoE: Advantage-Guided Preference Fusion for Learning Where to Explorecited by 0
  5. 2026Information-theoretic receding-horizon active learning of nonlinear dynamical systemscited by 0