mlsanityData to 7 Oct 2026

Exploration via Planning for Information about the Optimal Trajectory

arXiv 2210.04642 ↗First posted 6 Oct 2022cs.LG cs.AI cs.ROcited by 8

Not on the map. Why?

Abstract (arXiv)

Many potential applications of reinforcement learning (RL) are stymied by the large numbers of samples required to learn an effective policy. This is especially true when applying RL to real-world control tasks, e.g. in the sciences or robotics, where executing a policy in the environment is costly. In popular RL algorithms, agents typically explore either by adding stochasticity to a reward-maximizing policy or by attempting to gather maximal information about environment dynamics without taking the given task into account. In this work, we develop a method that allows us to plan for exploration while taking both the task and the current knowledge about the dynamics into account. The key insight to our approach is to plan an action sequence that maximizes the expected information gain about the optimal trajectory for the task at hand. We demonstrate that our method learns strong policies with 2x fewer samples than strong exploration baselines and 200x fewer samples than model free methods on a diverse set of low-to-medium dimensional control tasks in both the open-loop and closed-loop control settings.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Builds on 37 of its 84 references that are in the map. First posted on arXiv 6 Oct 2022.

Builds on

37 of 84 references.

References · 37

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  3. 2015Continuous control with deep reinforcement learningcited by 3,279
  4. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  5. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  6. 2016OpenAI Gymcited by 1,684
  7. 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,173
  8. 2018Exploration by Random Network Distillationcited by 966
  9. 2019When to Trust Your Model: Model-Based Policy Optimizationcited by 753
  10. 2018Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Modelscited by 748
  11. 2016Deep Exploration via Bootstrapped DQNcited by 518
  12. 2017A Tutorial on Thompson Samplingcited by 420
25 more
  1. 2020Planning to Explore via Self-Supervised World Modelscited by 365
  2. 2019Self-Supervised Exploration via Disagreementcited by 263
  3. 2014Predictive Entropy Search for Efficient Global Optimization of Black-box Functionscited by 201
  4. 2018Optimization of Molecules via Deep Reinforcement Learningcited by 199
  5. 2014Learning to Optimize via Information-Directed Samplingcited by 168
  6. 2017Deep Exploration via Randomized Value Functionscited by 155
  7. 2020SUNRISE: A Simple Unified Framework for Ensemble Learning in Deep Reinforcement Learningcited by 150
  8. 2019End-to-End Model-Free Reinforcement Learning for Urban Driving using Implicit Affordancescited by 147
  9. 2020Efficiently Sampling Functions from Gaussian Process Posteriorscited by 125
  10. 2019Model-Based Reinforcement Learning with a Generative Model is Minimax Optimalcited by 124
  11. 2018Model-Based Active Explorationcited by 105
  12. 2020Sample-efficient Cross-Entropy Method for Real-time Planningcited by 90
  13. 2020Efficient Model-Based Reinforcement Learning through Optimistic Policy Search and Planningcited by 83
  14. 2018Information-Directed Exploration for Deep Reinforcement Learningcited by 52
  15. 2017Data-Efficient Reinforcement Learning with Probabilistic Model Predictive Controlcited by 50
  16. 2020Reinforcement Learning through Active Inferencecited by 47
  17. 2021MBRL-Lib: A Modular Library for Model-based Reinforcement Learningcited by 41
  18. 2020Ready Policy One: World Building Through Active Learningcited by 33
  19. 2021Bayesian Algorithm Execution: Estimating Computable Properties of Black-box Functions Using Mutual Informationcited by 33
  20. 2013Model-Based Bayesian Explorationcited by 21
  21. 2020Neural Dynamical Systems: Balancing Structure and Flexibility in Physical Predictioncited by 18
  22. 2021An Experimental Design Perspective on Model-Based Reinforcement Learningcited by 16
  23. 2019Receding Horizon Curiositycited by 10
  24. 2021Anti-Concentrated Confidence Bonuses for Scalable Explorationcited by 7
  25. 2021Explore the Context: Optimal Data Collection for Context-Conditional Dynamics Modelscited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 1; 2024: 2; 2025: 3; 2026: 1.

8 papers.

  1. 2022Near-optimal Policy Identification in Active Reinforcement Learningcited by 7
  2. 2023PID-Inspired Inductive Biases for Deep Reinforcement Learning in Partially Observable Control Taskscited by 4
  3. 2024Sample-efficient Bayesian Optimisation Using Known Invariancescited by 4
  4. 2024Efficient Model-Based Reinforcement Learning Through Optimistic Thompson Samplingcited by 3
  5. 2025Neural Fidelity Calibration for Informative Sim-to-Real Adaptationcited by 1
  6. 2025Increasing Information for Model Predictive Control with Semi-Markov Decision Processescited by 0
  7. 2025HAEPO: History-Aggregated Exploratory Policy Optimizationcited by 0
  8. 2026CIG: Exploration via Conditional Information Gaincited by 0