State-Aware Proximal Pessimistic Algorithms for Offline Reinforcement Learning
On the map: Offline RL and imitation learning from data → Offline RL algorithms
Abstract (arXiv)
Pessimism is of great importance in offline reinforcement learning (RL). One broad category of offline RL algorithms fulfills pessimism by explicit or implicit behavior regularization. However, most of them only consider policy divergence as behavior regularization, ignoring the effect of how the offline state distribution differs with that of the learning policy, which may lead to under-pessimism for some states and over-pessimism for others. Taking account of this problem, we propose a principled algorithmic framework for offline RL, called State-Aware Proximal Pessimism (SA-PP). The key idea of SA-PP is leveraging discounted stationary state distribution ratios between the learning policy and the offline dataset to modulate the degree of behavior regularization in a state-wise manner, so that pessimism can be implemented in a more appropriate way. We first provide theoretical justifications on the superiority of SA-PP over previous algorithms, demonstrating that SA-PP produces a lower suboptimality upper bound in a broad range of settings. Furthermore, we propose a new algorithm named State-Aware Conservative Q-Learning (SA-CQL), by building SA-PP upon representative CQL algorithm with the help of DualDICE for estimating discounted stationary state distribution ratios. Extensive experiments on standard offline RL benchmark show that SA-CQL outperforms the popular baselines on a large portion of benchmarks and attains the highest average return.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 27 of its 38 references that are in the map. First posted on arXiv 28 Nov 2022.
Builds on
27 of 38 references.
Closest ancestors · 17
Same direction, newest first
- 2021Offline Reinforcement Learning with Soft Behavior Regularization14 Oct
- 2021Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemble4 Oct
- 2021Provable Benefits of Actor-Critic Methods for Offline Reinforcement Learning19 Aug
- 2021OptiDICE: Offline Policy Optimization via Stationary Distribution Correction Estimation21 Jun
- 2021Bellman-consistent Pessimism for Offline Reinforcement Learning13 Jun
- 2021A Minimalist Approach to Offline Reinforcement Learning12 Jun
- 2021Uncertainty Weighted Actor-Critic for Offline Reinforcement Learning17 May
- 2021Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimism22 Mar
- 2021Offline Reinforcement Learning with Fisher Divergence Critic Regularization14 Mar
- 2020Is Pessimism Provably Efficient for Offline RL?30 Dec
- 2020Provably Good Batch Reinforcement Learning Without Great Exploration16 Jul
- 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
5 more
- 2020MOPO: Model-based Offline Policy Optimization27 May
- 2020MOReL : Model-Based Offline Reinforcement Learning12 May
- 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
- 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr
- 2019Behavior Regularized Offline Reinforcement Learning26 Nov
Other references · 10
- 2015Continuous control with deep reinforcement learningcited by 3,407
- 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,653
- 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,241
- 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 840
- 2018Graph Convolutional Policy Network for Goal-Directed Molecular Graph Generationcited by 364
- 2019DualDICE: Behavior-Agnostic Estimation of Discounted Stationary Distribution Correctionscited by 254
- 2019AlgaeDICE: Policy Gradient from Arbitrary Experiencecited by 188
- 2019AVID: Learning Multi-Stage Tasks via Pixel-Level Translation of Human Videoscited by 105
- 2020The Importance of Pessimism in Fixed-Dataset Policy Optimizationcited by 99
- 2019MolecularRNN: Generating realistic molecular graphs with optimized propertiescited by 82
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2022Pessimism in the Face of Confounders: Provably Efficient Offline Reinforcement Learning in Partially Observable Markov Decision Processes
- 2021Dealing with the Unknown: Pessimistic Offline Reinforcement Learning
- 2023Double Pessimism is Provably Efficient for Distributionally Robust Offline Reinforcement Learning: Generic Algorithm and Robust Partial Coverage
- 2025Semi-pessimistic Reinforcement Learning
- 2026Generalization in offline RL: The structure is more important than the amount of pessimism
- 2022Pessimistic Q-Learning for Offline Reinforcement Learning: Towards Optimal Sample Complexity
- 2022Model-Based Offline Reinforcement Learning with Pessimism-Modulated Dynamics Belief
- 2024The Virtues of Pessimism in Inverse Reinforcement Learning