Coarse-Grained Smoothness for RL in Metric Spaces
Not on the map. Why?
Abstract (arXiv)
Principled decision-making in continuous state--action spaces is impossible without some assumptions. A common approach is to assume Lipschitz continuity of the Q-function. We show that, unfortunately, this property fails to hold in many typical domains. We propose a new coarse-grained smoothness definition that generalizes the notion of Lipschitz continuity, is more widely applicable, and allows us to compute significantly tighter bounds on Q-functions, leading to improved learning. We provide a theoretical analysis of our new smoothness definition, and discuss its implications and impact on control and exploration in continuous domains.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 17 of its 27 references that are in the map. First posted on arXiv 23 Oct 2021.
Builds on
17 of 27 references.
References · 17
- 2017Safe Model-based Reinforcement Learning with Stability Guaranteescited by 347
- 2015Norm-Based Capacity Control in Neural Networkscited by 299
- 2014Model-based Reinforcement Learning and the Eluder Dimensioncited by 116
- 2018Lipschitz Continuity in Model-based Reinforcement Learningcited by 90
- 2019Contextual Bandits with Continuous Actions: Smoothing, Zooming, and Adaptingcited by 42
- 2018Q-learning with Nearest Neighborscited by 37
- 2019Non-Stationary Markov Decision Processes, a Worst-Case Approach using Model-Based Reinforcement Learning, Extended versioncited by 36
- 2020Towards Safe Policy Improvement for Non-Stationary MDPscited by 26
- 2020Lipschitz Lifelong Reinforcement Learningcited by 23
- 2019Efficient Model-free Reinforcement Learning in Metric Spacescited by 18
- 2013Online Regret Bounds for Undiscounted Continuous Reinforcement Learningcited by 17
- 2019Combining Parametric and Nonparametric Models for Off-Policy Evaluationcited by 17
5 more
- 2018Improved robustness to adversarial examples using Lipschitz regularization of the losscited by 13
- 2020Adaptive Discretization for Model-Based Reinforcement Learningcited by 13
- 2019Learning to Control in Metric Space with Optimal Regretcited by 9
- 2020Zooming for Efficient Model-Free Reinforcement Learning in Metric Spacescited by 7
- 2020Off-Policy Interval Estimation with Lipschitz Value Iterationcited by 4
Built on by
Citing papers per year: 2022: 1; 2023: 1.
2 papers.