mlsanityData to 7 Oct 2026

Posterior Sampling for Deep Reinforcement Learning

arXiv 2305.00477 ↗First posted 30 Apr 2023cs.LG cs.AIcited by 16

Not on the map. Why?

Abstract (arXiv)

Despite remarkable successes, deep reinforcement learning algorithms remain sample inefficient: they require an enormous amount of trial and error to find good policies. Model-based algorithms promise sample efficiency by building an environment model that can be used for planning. Posterior Sampling for Reinforcement Learning is such a model-based algorithm that has attracted significant interest due to its performance in the tabular setting. This paper introduces Posterior Sampling for Deep Reinforcement Learning (PSDRL), the first truly scalable approximation of Posterior Sampling for Reinforcement Learning that retains its model-based essence. PSDRL combines efficient uncertainty quantification over latent state space models with a specially tailored continual planning algorithm based on value-function approximation. Extensive experiments on the Atari benchmark show that PSDRL significantly outperforms previous state-of-the-art attempts at scaling up posterior sampling while being competitive with a state-of-the-art (model-based) reinforcement learning method, both in sample efficiency and computational efficiency.

Key facts. Cited by 16 papers in the map (data through 7 Oct 2026). Builds on 41 of its 53 references that are in the map. First posted on arXiv 30 Apr 2023.

Builds on

41 of 53 references.

References · 41

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  3. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,598
  4. 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,327
  5. 2018Learning Latent Dynamics for Planning from Pixelscited by 1,221
  6. 2019Dota 2 with Large Scale Deep Reinforcement Learningcited by 1,084
  7. 2015Prioritized Experience Replaycited by 1,070
  8. 2020Mastering Atari with Discrete World Modelscited by 990
  9. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  10. 2018Recurrent World Models Facilitate Policy Evolutioncited by 761
  11. 2019When to Trust Your Model: Model-Based Policy Optimizationcited by 753
  12. 2018Diversity is All You Need: Learning Skills without a Reward Functioncited by 734
29 more
  1. 2016Unifying Count-Based Exploration and Intrinsic Motivationcited by 632
  2. 2016Deep Exploration via Bootstrapped DQNcited by 518
  3. 2017Minimax Regret Bounds for Reinforcement Learningcited by 500
  4. 2019Model-Based Reinforcement Learning for Ataricited by 495
  5. 2020Agent57: Outperforming the Atari Human Benchmarkcited by 300
  6. 2015Scalable Bayesian Optimization Using Deep Neural Networkscited by 284
  7. 2016Safe and Efficient Off-Policy Reinforcement Learningcited by 260
  8. 2020First return, then explorecited by 256
  9. 2013(More) Efficient Reinforcement Learning via Posterior Samplingcited by 233
  10. 2018Randomized Prior Functions for Deep Reinforcement Learningcited by 218
  11. 2020Never Give Up: Learning Directed Exploration Strategiescited by 210
  12. 2019Tighter Problem-Dependent Regret Bounds in Reinforcement Learning without Domain Knowledge using Value Function Boundscited by 194
  13. 2018Deep Bayesian Bandits Showdown: An Empirical Comparison of Bayesian Deep Networks for Thompson Samplingcited by 166
  14. 2017Deep Exploration via Randomized Value Functionscited by 155
  15. 2014Generalization and Exploration via Randomized Value Functionscited by 112
  16. 2018Episodic Curiosity through Reachabilitycited by 109
  17. 2018Playing hard exploration games by watching YouTubecited by 98
  18. 2017The Uncertainty Bellman Equation and Explorationcited by 81
  19. 2020DisCor: Corrective Feedback in Reinforcement Learning via Distribution Correctioncited by 71
  20. 2018Efficient Exploration through Bayesian Deep Q-Networkscited by 67
  21. 2019Worst-Case Regret Bounds for Exploration via Randomized Value Functionscited by 57
  22. 2021UCB Momentum Q-learning: Correcting the bias without forgettingcited by 45
  23. 2018Successor Uncertainties: Exploration and Uncertainty in Temporal Difference Learningcited by 35
  24. 2021Principled Exploration via Optimistic Bootstrapping and Backward Inductioncited by 26
  25. 2020Optimistic Exploration even with a Pessimistic Initialisationcited by 25
  26. 2022From Dirichlet to Rubin: Optimistic Exploration in RL without Bonusescited by 22
  27. 2020Model-based Reinforcement Learning for Continuous Control with Posterior Samplingcited by 21
  28. 2020Temporal Difference Uncertainties as a Signal for Explorationcited by 15
  29. 2022Hardness in Markov Decision Processes: Theory and Practicecited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 4; 2025: 8; 2026: 3.

16 papers.

Sort:

By area:

  1. 2025Toward Efficient Exploration by Large Language Model Agentscited by 15
  2. 2023Improved Bayesian Regret Bounds for Thompson Sampling in Reinforcement Learningcited by 7
  3. 2024Function-space Parameterization of Neural Networks for Sequential Learningcited by 7
  4. 2025The Sample Complexity of Online Reinforcement Learning: A Multi-model Perspectivecited by 5
  5. 2026Emergence of Exploration in Policy Gradient Reinforcement Learning via Retryingcited by 5
  6. 2025No-Regret Thompson Sampling for Finite-Horizon Markov Decision Processes with Gaussian Processescited by 3
  7. 2024Practical Bayesian Algorithm Execution via Posterior Samplingcited by 2
  8. 2025Provable Anytime Ensemble Sampling Algorithms in Nonlinear Contextual Banditscited by 2
  9. 2024Isoperimetry is All We Need: Langevin Posterior Sampling for RL with Sublinear Regretcited by 1
  10. 2025Q-learning with Posterior Samplingcited by 1
  11. 2025Foundation Models as World Models: A Foundational Study in Text-Based GridWorldscited by 1
  12. 2026Offline Policy Optimization with Posterior Samplingcited by 1
4 more
  1. 2024VDSC: Enhancing Exploration Timing with Value Discrepancy and State Countscited by 0
  2. 2025Exploration with Foundation Models: Capabilities, Limitations, and Hybrid Approachescited by 0
  3. 2025Real-DRL: Teach and Learn in Realitycited by 0
  4. 2026Efficient Geothermal Well-Control Optimization via Diffusion-Surrogate Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2018Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Models
  2. 2018Algorithmic Framework for Model-based Deep Reinforcement Learning with Theoretical Guarantees
  3. 2024On Sample-Efficient Offline Reinforcement Learning: Data Diversity, Posterior Sampling, and Beyond
  4. 2025Adversarial Diffusion for Robust Reinforcement Learning
  5. 2022MoDem: Accelerating Visual Model-Based Reinforcement Learning with Demonstrations
  6. 2025Sample and Computationally Efficient Continuous-Time Reinforcement Learning with General Function Approximation
  7. 2025Prior-Guided Diffusion Planning for Offline Reinforcement Learning
  8. 2026FASTER: Value-Guided Sampling for Fast RL