Q-Learning under Finite Model Uncertainty
Not on the map. Why?
Abstract (arXiv)
We propose a robust Q-learning algorithm for Markov decision processes under model uncertainty when each state-action pair is associated with a finite ambiguity set of candidate transition kernels. This finite-measure framework enables highly flexible, user-designed uncertainty models and goes beyond the common KL and Wasserstein ball formulations. We establish almost sure convergence of the learned Q-function to the robust optimum, and derive non-asymptotic high-probability error bounds that separate stochastic approximation error from transition-kernel estimation error. Finally, we show that Wasserstein ball and parametric ambiguity sets can be approximated by finite ambiguity sets, allowing our algorithm to be used as a generic solver beyond the finite setting.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 16 of its 67 references that are in the map. First posted on arXiv 5 Jul 2024.
Builds on
16 of 67 references.
References · 16
- 2015Deep Reinforcement Learning with Double Q-learningcited by 1,704
- 2017Stabilising Experience Replay for Deep Multi-Agent Reinforcement Learningcited by 114
- 2021Sample Complexity of Robust Reinforcement Learning with a Generative Modelcited by 82
- 2022Policy Gradient Method For Robust Reinforcement Learningcited by 67
- 2020Reinforcement Learning in Economics and Financecited by 57
- 2017Count-Based Exploration in Feature Space for Reinforcement Learningcited by 48
- 2018Double Deep Q-Learning for Optimal Executioncited by 39
- 2023A Finite Sample Complexity Bound for Distributionally Robust Q-learningcited by 35
- 2021Deep Hedging of Derivatives Using Reinforcement Learningcited by 30
- 2020Active Model Estimation in Markov Decision Processescited by 20
- 2021Reinforcement Learning for Mean Field Games, with Applications to Economicscited by 14
- 2020Distributionally Robust Markov Decision Processes and their Connection to Risk Measurescited by 12
4 more
- 2022Reinforcement Learning Algorithm for Mixed Mean Field Control Gamescited by 7
- 2023Bounding the Difference between the Values of Robust and Non-Robust Markov Decision Problemscited by 3
- 2018Robust Optimal Control Using Conditional RiskMappings in Infinite Horizoncited by 2
- 2022Markov Decision Processes under Model Uncertaintycited by 2
Built on by
Citing papers per year: 2025: 1; 2026: 1.
2 papers.