mlsanityData to 7 Oct 2026

Posterior Coreset Construction with Kernelized Stein Discrepancy for Model-Based Reinforcement Learning

arXiv 2206.01162 ↗First posted 2 Jun 2022cs.LG math.OC stat.MLcited by 7

Not on the map. Why?

Abstract (arXiv)

Model-based approaches to reinforcement learning (MBRL) exhibit favorable performance in practice, but their theoretical guarantees in large spaces are mostly restricted to the setting when transition model is Gaussian or Lipschitz, and demands a posterior estimate whose representational complexity grows unbounded with time. In this work, we develop a novel MBRL method (i) which relaxes the assumptions on the target transition model to belong to a generic family of mixture models; (ii) is applicable to large-scale training by incorporating a compression step such that the posterior estimate consists of a Bayesian coreset of only statistically significant past state-action pairs; and (iii) exhibits a sublinear Bayesian regret. To achieve these results, we adopt an approach based upon Stein's method, which, under a smoothness condition on the constructed posterior and target, allows distributional distance to be evaluated in closed form as the kernelized Stein discrepancy (KSD). The aforementioned compression step is then computed in terms of greedily retaining only those samples which are more than a certain KSD away from the previous model estimate. Experimentally, we observe that this approach is competitive with several state-of-the-art RL methodologies, and can achieve up-to 50 percent reduction in wall clock time in some continuous control environments.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Builds on 32 of its 61 references that are in the map. First posted on arXiv 2 Jun 2022.

Builds on

32 of 61 references.

References · 32

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  2. 2016OpenAI Gymcited by 1,684
  3. 2019When to Trust Your Model: Model-Based Policy Optimizationcited by 753
  4. 2018Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Modelscited by 748
  5. 2019Provably Efficient Reinforcement Learning with Linear Function Approximationcited by 514
  6. 2018Is Q-learning Provably Efficient?cited by 485
  7. 2017Neural Network Dynamics for Model-Based Deep Reinforcement Learning with Model-Free Fine-Tuningcited by 288
  8. 2016A Kernelized Stein Discrepancy for Goodness-of-fit Tests and Model Evaluationcited by 236
  9. 2013(More) Efficient Reinforcement Learning via Posterior Samplingcited by 233
  10. 2020Model-Based Reinforcement Learning with Value-Targeted Regressioncited by 233
  11. 2019Reinforcement Learning in Feature Space: Matrix Bandit, Kernels, and Regret Boundcited by 199
  12. 2020Nearly Minimax Optimal Reinforcement Learning for Linear Mixture Markov Decision Processescited by 199
20 more
  1. 2018Distributed Distributional Deterministic Policy Gradientscited by 195
  2. 2020Learning Near Optimal Policies with Low Inherent Bellman Errorcited by 180
  3. 2017Unifying PAC and Regret: Uniform PAC Bounds for Episodic Reinforcement Learningcited by 175
  4. 2019Benchmarking Model-Based Reinforcement Learningcited by 158
  5. 2014Model-based Reinforcement Learning and the Eluder Dimensioncited by 116
  6. 2016Why is Posterior Sampling Better than Optimism for Reinforcement Learning?cited by 116
  7. 2015Gaussian Processes for Data-Efficient Learning in Robotics and Controlcited by 97
  8. 2015Measuring Sample Quality with Stein's Methodcited by 94
  9. 2018Bayesian Coreset Construction via Greedy Iterative Geodesic Ascentcited by 68
  10. 2017Automated Scalable Bayesian Inference via Hilbert Coresetscited by 55
  11. 2017Data-Efficient Reinforcement Learning with Probabilistic Model Predictive Controlcited by 50
  12. 2019On Online Learning in Kernelized Markov Decision Processescited by 32
  13. 2018Variational Bayesian Reinforcement Learning with Regret Boundscited by 31
  14. 2019Stein Point Markov Chain Monte Carlocited by 29
  15. 2020Online Model Selection for Reinforcement Learning with Function Approximationcited by 28
  16. 2020Model-based Reinforcement Learning for Continuous Control with Posterior Samplingcited by 21
  17. 2020Risk-Averse Learning by Temporal Difference Methodscited by 13
  18. 2020Consistent Online Gaussian Process Regression Without the Sample Complexity Bottleneckcited by 10
  19. 2020A Distributional Analysis of Sampling-Based Reinforcement Learning Algorithmscited by 3
  20. 2022Online, Informative MCMC Thinning with Kernelized Stein Discrepancycited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 4; 2024: 2; 2025: 0; 2026: 1.

7 papers.

  1. 2023PARL: A Unified Framework for Policy Alignment in Reinforcement Learning from Human Feedbackcited by 38
  2. 2023Improved Bayesian Regret Bounds for Thompson Sampling in Reinforcement Learningcited by 7
  3. 2023STEERING: Stein Information Directed Exploration for Model-Based Reinforcement Learningcited by 4
  4. 2024FedCore: Straggler-Free Federated Learning with Distributed Coresetscited by 2
  5. 2023Policy Gradient with Kernel Quadraturecited by 1
  6. 2026SteinGate: Tail-Sensitive Safe Reinforcement Learning via Stein Discrepancycited by 1
  7. 2024A resource-constrained stochastic scheduling algorithm for homeless street outreach and gleaning edible foodcited by 0