mlsanityData to 14 Oct 2026

Optimistic Active Exploration of Dynamical Systems

arXiv 2306.12371 ↗First posted 21 Jun 2023cs.LG cs.RO cs.SYcited by 32

Not on the map. Why?

Abstract (arXiv)

Reinforcement learning algorithms commonly seek to optimize policies for solving one particular task. How should we explore an unknown dynamical system such that the estimated model globally approximates the dynamics and allows us to solve multiple downstream tasks in a zero-shot manner? In this paper, we address this challenge, by developing an algorithm -- OPAX -- for active exploration. OPAX uses well-calibrated probabilistic models to quantify the epistemic uncertainty about the unknown dynamics. It optimistically -- w.r.t. to plausible dynamics -- maximizes the information gain between the unknown dynamics and state observations. We show how the resulting optimization problem can be reduced to an optimal control problem that can be solved at each episode using standard approaches. We analyze our algorithm for general models, and, in the case of Gaussian process dynamics, we give a first-of-its-kind sample complexity bound and show that the epistemic uncertainty converges to zero. In our experiments, we compare OPAX with other heuristic active exploration approaches on several environments. Our experiments show that OPAX is not only theoretically sound but also performs well for zero-shot planning on novel downstream tasks.

Key facts. Cited by 32 papers in the map (data through 14 Oct 2026). Builds on 42 of its 72 references that are in the map. First posted on arXiv 21 Jun 2023.

Builds on

42 of 72 references.

References · 42

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  2. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,926
  3. 2016OpenAI Gymcited by 1,736
  4. 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,194
  5. 2018DeepMind Control Suitecited by 956
  6. 2017A Survey on Multi-Task Learningcited by 840
  7. 2018Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Modelscited by 774
  8. 2018Diversity is All You Need: Learning Skills without a Reward Functioncited by 752
  9. 2018Accurate Uncertainties for Deep Learning Using Calibrated Regressioncited by 422
  10. 2020Planning to Explore via Self-Supervised World Modelscited by 368
  11. 2017Safe Model-based Reinforcement Learning with Stability Guaranteescited by 366
  12. 2017On Kernelized Multi-armed Banditscited by 352
30 more
  1. 2019Self-Supervised Exploration via Disagreementcited by 265
  2. 2020Reward-Free Exploration for Reinforcement Learningcited by 177
  3. 2018Learning Without Mixing: Towards A Sharp Analysis of Linear System Identificationcited by 158
  4. 2020On Information Gain and Regret Bounds in Gaussian Process Banditscited by 158
  5. 2015Incentivizing Exploration In Reinforcement Learning With Deep Predictive Modelscited by 154
  6. 2020Naive Exploration is Optimal for Online LQRcited by 116
  7. 2018Model-Based Active Explorationcited by 105
  8. 2020Sample-efficient Cross-Entropy Method for Real-time Planningcited by 95
  9. 2020Information Theoretic Regret Bounds for Online Nonlinear Controlcited by 90
  10. 2020Efficient Model-Based Reinforcement Learning through Optimistic Policy Search and Planningcited by 86
  11. 2020Active Learning for Nonlinear System Identification with Guaranteescited by 80
  12. 2020Adaptive Reward-Free Explorationcited by 77
  13. 2019Nonlinear System Identification: A User-Oriented Roadmapcited by 57
  14. 2022Constrained Policy Optimization via Bayesian World Modelscited by 57
  15. 2019Towards Practical Multi-Object Manipulation using Relational Reinforcement Learningcited by 55
  16. 2022Reward-Free RL is No Harder Than Reward-Aware RL in Linear Markov Decision Processescited by 49
  17. 2021Efficient Model-Based Multi-Agent Mean-Field Reinforcement Learningcited by 38
  18. 2020Active Learning for Identification of Linear Dynamical Systemscited by 33
  19. 2022Learning with little mixingcited by 33
  20. 2022Curious Exploration via Structured World Models Yields Zero-Shot Object Manipulationcited by 31
  21. 2022On the Statistical Efficiency of Reward-Free Exploration in Non-Linear RLcited by 27
  22. 2019Actively Learning Gaussian Process Dynamicscited by 26
  23. 2022Single Trajectory Nonparametric Learning of Nonlinear Dynamicscited by 26
  24. 2022Model-based Causal Bayesian Optimizationcited by 26
  25. 2020Active Model Estimation in Markov Decision Processescited by 20
  26. 2021On Reward-Free RL with Kernel and Neural Function Approximations: Single-Agent MDP and Markov Gamecited by 20
  27. 2022Active Exploration via Experiment Design in Markov Chainscited by 19
  28. 2023Optimal Exploration for Model-Based RL in Nonlinear Systemscited by 18
  29. 2021An Experimental Design Perspective on Model-Based Reinforcement Learningcited by 16
  30. 2023Hallucinated Adversarial Control for Conservative Offline Policy Evaluationcited by 13

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 12; 2025: 9; 2026: 10.

32 papers.

Sort:

By area:

  1. 2024MaxInfoRL: Boosting exploration in reinforcement learning through information gain maximizationcited by 30
  2. 2024ActSafe: Active Exploration with Safety Constraints for Reinforcement Learningcited by 23
  3. 2025DISCOVER: Automated Curricula for Sparse-Reward Reinforcement Learningcited by 12
  4. 2024When to Sense and Control? A Time-adaptive Approach for Continuous-Time RLcited by 9
  5. 2024Bridging the Sim-to-Real Gap with Bayesian Inferencecited by 8
  6. 2024All models are wrong, some are useful: Model Selection with Limited Labelscited by 8
  7. 2025Epistemically-guided forward-backward explorationcited by 8
  8. 2024Active Learning for Control-Oriented Identification of Nonlinear Systemscited by 7
  9. 2025SOMBRL: Scalable and Optimistic Model-Based RLcited by 7
  10. 2026LLM-ACES: Closed-Loop Discovery of Dynamical Systems with LLM-Guided Adaptive Searchcited by 7
  11. 2023Data-Efficient Task Generalization via Probabilistic Model-based Meta Reinforcement Learningcited by 6
  12. 2024Beyond Optimism: Exploration With Partially Observable Rewardscited by 6
20 more
  1. 2026Safe Exploration via Policy Priorscited by 6
  2. 2026Biased Dreams: Limitations to Epistemic Uncertainty Quantification in Latent Dynamics Modelscited by 5
  3. 2024Safe Exploration Using Bayesian World Models and Log-Barrier Optimizationcited by 4
  4. 2024NeoRL: Efficient Exploration for Nonepisodic RLcited by 4
  5. 2024On Foundation Models for Dynamical Systems from Purely Synthetic Datacited by 4
  6. 2025Skill Learning via Policy Diversity Yields Identifiable Representations for Reinforcement Learningcited by 4
  7. 2024First, Learn What You Don't Know: Active Information Gathering for Driving at the Limits of Handlingcited by 3
  8. 2025Sample-efficient and Scalable Exploration in Continuous-Time RLcited by 3
  9. 2025Learning Soft Robotic Dynamics with Active Explorationcited by 2
  10. 2025Neural Fidelity Calibration for Informative Sim-to-Real Adaptationcited by 1
  11. 2026Model-Based Reinforcement Learning for Control under Time-Varying Dynamicscited by 1
  12. 2024Active Learning of Dynamics Using Prior Domain Knowledge in the Sampling Processcited by 0
  13. 2025Logarithmic Regret for Nonlinear Controlcited by 0
  14. 2025Simulation Priors for Data-Efficient Deep Learningcited by 0
  15. 2026Generalized Information Gathering Under Dynamics Uncertaintycited by 0
  16. 2026Sampling-Based Safe Reinforcement Learningcited by 0
  17. 2026UBP2: Uncertainty-Balanced Preference Planning for Efficient Preference-based Reinforcement Learningcited by 0
  18. 2026Diagnose, Recover, Certify: Task Readiness under Hidden Dynamics Changescited by 0
  19. 2026Information-theoretic receding-horizon active learning of nonlinear dynamical systemscited by 0
  20. 2026Tactile Curiosity Drives Robot Interactioncited by 0