STEEL: Singularity-aware Reinforcement Learning
On the map: Offline RL and imitation learning from data → Offline RL algorithms
Abstract (arXiv)
Batch reinforcement learning (RL) aims at leveraging pre-collected data to find an optimal policy that maximizes the expected total rewards in a dynamic environment. The existing methods require absolutely continuous assumption (e.g., there do not exist non-overlapping regions) on the distribution induced by target policies with respect to the data distribution over either the state or action or both. We propose a new batch RL algorithm that allows for singularity for both state and action spaces (e.g., existence of non-overlapping regions between offline data distribution and the distribution induced by the target policies) in the setting of an infinite-horizon Markov decision process with continuous states and actions. We call our algorithm STEEL: SingulariTy-awarE rEinforcement Learning. Our algorithm is motivated by a new error analysis on off-policy evaluation, where we use maximum mean discrepancy, together with distributionally robust optimization, to characterize the error of off-policy evaluation caused by the possible singularity and to enable model extrapolation. By leveraging the idea of pessimism and under some technical conditions, we derive a first finite-sample regret guarantee for our proposed algorithm under singularity. Compared with existing algorithms,by requiring only minimal data-coverage assumption, STEEL improves the applicability and robustness of batch RL. In addition, a two-step adaptive STEEL, which is nearly tuning-free, is proposed. Extensive simulation studies and one (semi)-real experiment on personalized pricing demonstrate the superior performance of our methods in dealing with possible singularity in batch RL.
Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 27 of its 69 references that are in the map. First posted on arXiv 30 Jan 2023.
Builds on
27 of 69 references.
Closest ancestors · 11
Same direction, newest first
- 2022Offline Reinforcement Learning with Instrumental Variables in Confounded Markov Decision Processes18 Sep
- 2022Pessimistic Bootstrapping for Uncertainty-Driven Offline Reinforcement Learning23 Feb
- 2022Offline Reinforcement Learning with Realizability and Single-policy Concentrability9 Feb
- 2021Provable Benefits of Actor-Critic Methods for Offline Reinforcement Learning19 Aug
- 2021Bellman-consistent Pessimism for Offline Reinforcement Learning13 Jun
- 2021Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimism22 Mar
- 2021Instabilities of Offline RL with Pre-Trained Neural Representation8 Mar
- 2020Is Pessimism Provably Efficient for Offline RL?30 Dec
- 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
- 2020Doubly Robust Off-Policy Value and Gradient Estimation for Deterministic Policies6 Jun
- 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
Other references · 16
- 2015Continuous control with deep reinforcement learningcited by 3,279
- 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
- 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
- 2017Nonparametric regression using deep neural networks with ReLU activation functioncited by 431
- 2015Supersizing Self-supervision: Learning to Grasp from 50K Tries and 700 Robot Hourscited by 208
- 2020Batch Policy Learning in Average Reward Markov Decision Processescited by 69
- 2020Statistical Inference of the Value Function for Reinforcement Learning in Infinite Horizon Settingscited by 65
- 2018Policy Evaluation and Optimization with Continuous Treatmentscited by 57
- 2019Meta Dynamic Pricing: Transfer Learning Across Experimentscited by 41
- 2021Deeply-Debiased Off-Policy Interval Estimationcited by 36
- 2022On Well-posedness and Minimax Optimal Rates of Nonparametric Q-function Estimation in Off-policy Evaluationcited by 33
- 2020Learning Optimal Distributionally Robust Individualized Treatment Rulescited by 28
4 more
- 2019Semi-Parametric Efficient Policy Learning with Continuous Actionscited by 26
- 2018PAC-Bayes bounds for stable algorithms with instance-dependent priorscited by 21
- 2020Distributionally Robust Batch Contextual Banditscited by 20
- 2019On Robustness of Individualized Decision Rulescited by 6
Built on by
Citing papers per year: 2023: 1; 2024: 1; 2025: 1.
3 papers.
- 2024Dual Active Learning for Reinforcement Learning from Human Feedbackcited by 19
- 2023Stackelberg Batch Policy Learningcited by 1
- 2025Generalized Fitted Q-Iteration with Clustered Datacited by 0
- 2025Generalized Fitted Q-Iteration with Clustered Data4 Oct 2025
- 2024Dual Active Learning for Reinforcement Learning from Human Feedback3 Oct 2024
- 2023Stackelberg Batch Policy Learning28 Sep 2023
Similar, not cited
Close in wording, no citation either way
- 2020Provably Good Batch Reinforcement Learning Without Great Exploration
- 2021Continuous Doubly Constrained Batch Reinforcement Learning
- 2024Iterative Batch Reinforcement Learning via Safe Diversified Model-based Policy Search
- 2021Comparison and Unification of Three Regularization Methods in Batch Reinforcement Learning
- 2024On Sample-Efficient Offline Reinforcement Learning: Data Diversity, Posterior Sampling, and Beyond
- 2021Policy Finetuning: Bridging Sample-Efficient Offline and Online Reinforcement Learning
- 2020BRPO: Batch Residual Policy Optimization
- 2022Settling the Sample Complexity of Model-Based Offline Reinforcement Learning