Utilizing Maximum Mean Discrepancy Barycenter for Propagating the Uncertainty of Value Functions in Reinforcement Learning
Not on the map. Why?
Abstract (arXiv)
Accounting for the uncertainty of value functions boosts exploration in Reinforcement Learning (RL). Our work introduces Maximum Mean Discrepancy Q-Learning (MMD-QL) to improve Wasserstein Q-Learning (WQL) for uncertainty propagation during Temporal Difference (TD) updates. MMD-QL uses the MMD barycenter for this purpose, as MMD provides a tighter estimate of closeness between probability measures than the Wasserstein distance. Firstly, we establish that MMD-QL is Probably Approximately Correct in MDP (PAC-MDP) under the average loss metric. Concerning the accumulated rewards, experiments on tabular environments show that MMD-QL outperforms WQL and other algorithms. Secondly, we incorporate deep networks into MMD-QL to create MMD Q-Network (MMD-QN). Making reasonable assumptions, we analyze the convergence rates of MMD-QN using function approximation. Empirical results on challenging Atari games demonstrate that MMD-QN performs well compared to benchmark deep RL algorithms, highlighting its effectiveness in handling large state-action spaces.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 28 of its 51 references that are in the map. First posted on arXiv 31 Mar 2024.
Builds on
28 of 51 references.
References · 28
- 2015Deep Reinforcement Learning with Double Q-learningcited by 1,704
- 2015Dueling Network Architectures for Deep Reinforcement Learningcited by 769
- 2016Deep Exploration via Bootstrapped DQNcited by 518
- 2017Minimax Regret Bounds for Reinforcement Learningcited by 500
- 2018Is Q-learning Provably Efficient?cited by 485
- 2017Nonparametric regression using deep neural networks with ReLU activation functioncited by 431
- 2017Revisiting the Arcade Learning Environment: Evaluation Protocols and Open Problems for General Agentscited by 267
- 2013(More) Efficient Reinforcement Learning via Posterior Samplingcited by 233
- 2020Model-Based Reinforcement Learning with Value-Targeted Regressioncited by 233
- 2018Randomized Prior Functions for Deep Reinforcement Learningcited by 218
- 2019A Theoretical Analysis of Deep Q-Learningcited by 212
- 2019Tighter Problem-Dependent Regret Bounds in Reinforcement Learning without Domain Knowledge using Value Function Boundscited by 194
16 more
- 2017Unifying PAC and Regret: Uniform PAC Bounds for Episodic Reinforcement Learningcited by 175
- 2021Epistemic Neural Networkscited by 120
- 2016Why is Posterior Sampling Better than Optimism for Reinforcement Learning?cited by 116
- 2014Generalization and Exploration via Randomized Value Functionscited by 112
- 2015Sample Complexity of Episodic Fixed-Horizon Reinforcement Learningcited by 105
- 2020Is Reinforcement Learning More Difficult Than Bandits? A Near-optimal Algorithm Escaping the Curse of Horizoncited by 99
- 2019Model-free Reinforcement Learning in Infinite-horizon Average-reward Markov Decision Processescited by 81
- 2018Efficient Exploration through Bayesian Deep Q-Networkscited by 67
- 2018Efficient Bias-Span-Constrained Exploration-Exploitation in Reinforcement Learningcited by 65
- 2020MushroomRL: Simplifying Reinforcement Learning Researchcited by 50
- 2019Statistical Inference for Generative Models with Maximum Mean Discrepancycited by 43
- 2022Horizon-Free Reinforcement Learning in Polynomial Time: the Power of Stationary Policiescited by 27
- 2020Tightening Exploration in Upper Confidence Reinforcement Learningcited by 24
- 2020Estimating Barycenters of Measures in High Dimensionscited by 18
- 2023Sharp Variance-Dependent Bounds in Reinforcement Learning: Best of Both Worlds in Stochastic and Deterministic Environmentscited by 17
- 2021Controlling Wasserstein Distances by Kernel Norms with Application to Compressive Statistical Learningcited by 13
Built on by
Not cited yet.