mlsanityData to 14 Oct 2026

Grid-Mapping Pseudo-Count Constraint for Offline Reinforcement Learning

arXiv 2404.02545 ↗First posted 3 Apr 2024cs.LG cs.AIcited by 1

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Offline reinforcement learning learns from a static dataset without interacting with environments, which ensures security and thus owns a good application prospect. However, directly applying naive reinforcement learning algorithm usually fails in an offline environment due to inaccurate Q value approximation caused by out-of-distribution (OOD) state-actions. It is an effective way to solve this problem by penalizing the Q-value of OOD state-actions. Among the methods of punishing OOD state-actions, count-based methods have achieved good results in discrete domains in a simple form. Inspired by it, a novel pseudo-count method for continuous domains called Grid-Mapping Pseudo-Count method (GPC) is proposed by extending the count-based method from discrete to continuous domains. Firstly, the continuous state and action space are mapped to discrete space using Grid-Mapping, then the Q-values of OOD state-actions are constrained through pseudo-count. Secondly, the theoretical proof is given to show that GPC can obtain appropriate uncertainty constraints under fewer assumptions than other pseudo-count methods. Thirdly, GPC is combined with Soft Actor-Critic algorithm (SAC) to get a new algorithm called GPC-SAC. Lastly, experiments on D4RL datasets are given to show that GPC-SAC has better performance and less computational cost than other algorithms that constrain the Q-value.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 39 of its 51 references that are in the map. First posted on arXiv 3 Apr 2024.

Builds on

39 of 51 references.

Closest ancestors · 17

Same direction, newest first

  1. 2023Model-based Offline Reinforcement Learning with Count-based Conservatism21 Jul
  2. 2022Confidence-Conditioned Value Functions for Offline Reinforcement Learning8 Dec
  3. 2022A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problems2 Mar
  4. 2022Pessimistic Bootstrapping for Uncertainty-Driven Offline Reinforcement Learning23 Feb
  5. 2021Offline Reinforcement Learning with Implicit Q-Learning12 Oct
  6. 2021Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemble4 Oct
  7. 2021A Minimalist Approach to Offline Reinforcement Learning12 Jun
  8. 2021Offline Reinforcement Learning as Anti-Exploration11 Jun
  9. 2021Uncertainty Weighted Actor-Critic for Offline Reinforcement Learning17 May
  10. 2021COMBO: Conservative Offline Model-Based Policy Optimization16 Feb
  11. 2020Is Pessimism Provably Efficient for Offline RL?30 Dec
  12. 2020On Reward-Free Reinforcement Learning with Linear Function Approximation19 Jun
5 more
  1. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  2. 2020MOPO: Model-based Offline Policy Optimization27 May
  3. 2020MOReL : Model-Based Offline Reinforcement Learning12 May
  4. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
  5. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr

Other references · 22

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  2. 2015Trust Region Policy Optimizationcited by 2,580
  3. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,241
  4. 2019Dota 2 with Large Scale Deep Reinforcement Learningcited by 1,103
  5. 2016Conditional Image Generation with PixelCNN Decoderscited by 862
  6. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 840
  7. 2019When to Trust Your Model: Model-Based Policy Optimizationcited by 773
  8. 2016Unifying Count-Based Exploration and Intrinsic Motivationcited by 639
  9. 2016Deep Exploration via Bootstrapped DQNcited by 527
  10. 2019Provably Efficient Reinforcement Learning with Linear Function Approximationcited by 520
  11. 2018Regularisation of Neural Networks by Enforcing Lipschitz Continuitycited by 314
  12. 2019Reinforcement Learning in Healthcare: A Surveycited by 310
10 more
  1. 2016#Exploration: A Study of Count-Based Exploration for Deep Reinforcement Learningcited by 309
  2. 2017Count-Based Exploration with Neural Density Modelscited by 257
  3. 2020FLAMBE: Structural Complexity and Representation Learning of Low Rank MDPscited by 233
  4. 2018Model-Ensemble Trust-Region Policy Optimizationcited by 219
  5. 2018Randomized Prior Functions for Deep Reinforcement Learningcited by 219
  6. 2019Estimating Risk and Uncertainty in Deep Reinforcement Learningcited by 70
  7. 2022Sample Efficient Deep Reinforcement Learning via Uncertainty Estimationcited by 27
  8. 2018Contingency-Aware Exploration in Reinforcement Learningcited by 26
  9. 2020Reinforcement Learning with Uncertainty Estimation for Tactical Decision-Making in Intersectionscited by 15
  10. 2020Multi-Agent Safe Planning with Gaussian Processescited by 13

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Efficient Anti-exploration via VQVAE and Fuzzy Clustering in Offline Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2022Mildly Conservative Q-Learning for Offline Reinforcement Learning
  2. 2023DOMAIN: MilDly COnservative Model-BAsed OfflINe Reinforcement Learning
  3. 2021The Least Restriction for Offline Reinforcement Learning
  4. 2023Conservative State Value Estimation for Offline Reinforcement Learning
  5. 2024SelfBC: Self Behavior Cloning for Offline Reinforcement Learning
  6. 2025Beyond Non-Expert Demonstrations: Outcome-Driven Action Constraint for Offline Reinforcement Learning
  7. 2021Constraints Penalized Q-learning for Safe Offline Reinforcement Learning
  8. 2021Offline RL Without Off-Policy Evaluation