mlsanityData to 14 Oct 2026

STEERING: Stein Information Directed Exploration for Model-Based Reinforcement Learning

arXiv 2301.12038 ↗First posted 28 Jan 2023cs.LG cs.AI stat.MLcited by 4

Not on the map. Why?

Abstract (arXiv)

Directed Exploration is a crucial challenge in reinforcement learning (RL), especially when rewards are sparse. Information-directed sampling (IDS), which optimizes the information ratio, seeks to do so by augmenting regret with information gain. However, estimating information gain is computationally intractable or relies on restrictive assumptions which prohibit its use in many practical instances. In this work, we posit an alternative exploration incentive in terms of the integral probability metric (IPM) between a current estimate of the transition model and the unknown optimal, which under suitable conditions, can be computed in closed form with the kernelized Stein discrepancy (KSD). Based on KSD, we develop a novel algorithm algo: STEin information dirEcted exploration for model-based Reinforcement LearnING. To enable its derivation, we develop fundamentally new variants of KSD for discrete conditional distributions. We further establish that algo archives sublinear Bayesian regret, improving upon prior learning rates of information-augmented MBRL. Experimentally, we show that the proposed algorithm is computationally affordable and outperforms several prior approaches.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 50 of its 86 references that are in the map. First posted on arXiv 28 Jan 2023.

Builds on

50 of 86 references.

References · 50

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  3. 2015Continuous control with deep reinforcement learningcited by 3,407
  4. 2013Playing Atari with Deep Reinforcement Learningcited by 3,274
  5. 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,194
  6. 2018Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Modelscited by 774
  7. 2019When to Trust Your Model: Model-Based Policy Optimizationcited by 773
  8. 2019Provably Efficient Reinforcement Learning with Linear Function Approximationcited by 520
  9. 2018Is Q-learning Provably Efficient?cited by 487
  10. 2017A Tutorial on Thompson Samplingcited by 433
  11. 2018Large-Scale Study of Curiosity-Driven Learningcited by 280
  12. 2019Self-Supervised Exploration via Disagreementcited by 265
38 more
  1. 2013Learning to Optimize Via Posterior Samplingcited by 256
  2. 2017Non-convex learning via Stochastic Gradient Langevin Dynamics: a nonasymptotic analysiscited by 243
  3. 2016A Kernelized Stein Discrepancy for Goodness-of-fit Tests and Model Evaluationcited by 236
  4. 2020Model-Based Reinforcement Learning with Value-Targeted Regressioncited by 234
  5. 2013(More) Efficient Reinforcement Learning via Posterior Samplingcited by 233
  6. 2019Reinforcement Learning in Feature Space: Matrix Bandit, Kernels, and Regret Boundcited by 201
  7. 2014An Information-Theoretic Analysis of Thompson Samplingcited by 178
  8. 2018Understanding the impact of entropy on policy optimizationcited by 176
  9. 2021Maximum Entropy RL (Provably) Solves Some Robust RL Problemscited by 176
  10. 2014Learning to Optimize via Information-Directed Samplingcited by 168
  11. 2017Deep Exploration via Randomized Value Functionscited by 156
  12. 2016Why is Posterior Sampling Better than Optimism for Reinforcement Learning?cited by 117
  13. 2014Model-based Reinforcement Learning and the Eluder Dimensioncited by 116
  14. 2018Model-Based Active Explorationcited by 105
  15. 2019Frequentist Regret Bounds for Randomized Least-Squares Value Iterationcited by 101
  16. 2015Measuring Sample Quality with Stein's Methodcited by 94
  17. 2017Surprise-Based Intrinsic Motivation for Deep Reinforcement Learningcited by 83
  18. 2017The Uncertainty Bellman Equation and Explorationcited by 82
  19. 2019Wasserstein Dependency Measure for Representation Learningcited by 69
  20. 2021Reinforcement Learning, Bit by Bitcited by 59
  21. 2022Reinforcement Learning with Sparse Rewards using Guidance from Offline Demonstrationcited by 54
  22. 2019Information-Theoretic Confidence Bounds for Reinforcement Learningcited by 51
  23. 2019An Information-Theoretic Approach to Minimax Regret in Partial Monitoringcited by 42
  24. 2019On Online Learning in Kernelized Markov Decision Processescited by 35
  25. 2019Stein Point Markov Chain Monte Carlocited by 29
  26. 2022Regret Bounds for Information-Directed Reinforcement Learningcited by 24
  27. 2019Connections Between Mirror Descent, Thompson Sampling and the Information Ratiocited by 23
  28. 2020Model-based Reinforcement Learning for Continuous Control with Posterior Samplingcited by 21
  29. 2020Testing Goodness of Fit of Conditional Density Models with Kernelscited by 19
  30. 2022On the Hidden Biases of Policy Mirror Ascent in Continuous Action Spacescited by 11
  31. 2017Learning-based Control of Unknown Linear Systems with Thompson Samplingcited by 10
  32. 2020Consistent Online Gaussian Process Regression Without the Sample Complexity Bottleneckcited by 10
  33. 2022Posterior Coreset Construction with Kernelized Stein Discrepancy for Model-Based Reinforcement Learningcited by 7
  34. 2020A Distributional Analysis of Sampling-Based Reinforcement Learning Algorithmscited by 3
  35. 2022Online, Informative MCMC Thinning with Kernelized Stein Discrepancycited by 3
  36. 2019Practical Calculation of Gittins Indices for Multi-armed Banditscited by 2
  37. 2022Dealing with Sparse Rewards in Continuous Control Robotics via Heavy-Tailed Policiescited by 2
  38. 2022HTRON:Efficient Outdoor Navigation with Sparse Rewards via Heavy Tailed Adaptive Reinforce Algorithmcited by 2

Built on by

Cited by year

Citing papers per year: 2023: 1; 2024: 2; 2025: 1.

4 papers.

  1. 2024Value of Information and Reward Specification in Active Inference and POMDPscited by 4
  2. 2024Provably Efficient Information-Directed Sampling Algorithms for Multi-Agent Reinforcement Learningcited by 2
  3. 2023REBEL: Reward Regularization-Based Approach for Robotic Reinforcement Learning from Human Feedbackcited by 0
  4. 2025Maximum Total Correlation Reinforcement Learningcited by 0