mlsanityData to 7 Oct 2026

Efficient Anti-exploration via VQVAE and Fuzzy Clustering in Offline Reinforcement Learning

arXiv 2602.07889 ↗First posted 8 Feb 2026cs.LGcited by 0

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Pseudo-count is an effective anti-exploration method in offline reinforcement learning (RL) by counting state-action pairs and imposing a large penalty on rare or unseen state-action pair data. Existing anti-exploration methods count continuous state-action pairs by discretizing these data, but often suffer from the issues of dimension disaster and information loss in the discretization process, leading to efficiency and performance reduction, and even failure of policy learning. In this paper, a novel anti-exploration method based on Vector Quantized Variational Autoencoder (VQVAE) and fuzzy clustering in offline RL is proposed. We first propose an efficient pseudo-count method based on the multi-codebook VQVAE to discretize state-action pairs, and design an offline RL anti-exploitation method based on the proposed pseudo-count method to handle the dimension disaster issue and improve the learning efficiency. In addition, a codebook update mechanism based on fuzzy C-means (FCM) clustering is developed to improve the use rate of vectors in codebooks, addressing the information loss issue in the discretization process. The proposed method is evaluated on the benchmark of Datasets for Deep Data-Driven Reinforcement Learning (D4RL), and experimental results show that the proposed method performs better and requires less computing cost in multiple complex tasks compared to state-of-the-art (SOTA) methods.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 27 of its 42 references that are in the map. First posted on arXiv 8 Feb 2026.

Builds on

27 of 42 references.

Closest ancestors · 14

Same direction, newest first

  1. 2024Doubly Mild Generalization for Offline Reinforcement Learning12 Nov
  2. 2024Grid-Mapping Pseudo-Count Constraint for Offline Reinforcement Learning3 Apr
  3. 2024ODICE: Revealing the Mystery of Distribution Correction Estimation via Orthogonal-gradient Update1 Feb
  4. 2023Model-based Offline Reinforcement Learning with Count-based Conservatism21 Jul
  5. 2023Anti-Exploration by Random Network Distillation31 Jan
  6. 2022Mildly Conservative Q-Learning for Offline Reinforcement Learning9 Jun
  7. 2022A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problems2 Mar
  8. 2022The Challenges of Exploration for Offline Reinforcement Learning27 Jan
  9. 2021Offline Reinforcement Learning with Implicit Q-Learning12 Oct
  10. 2021Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemble4 Oct
  11. 2021A Minimalist Approach to Offline Reinforcement Learning12 Jun
  12. 2021Offline Reinforcement Learning as Anti-Exploration11 Jun
2 more
  1. 2020PLAS: Latent Action Space for Offline Reinforcement Learning14 Nov
  2. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May

Other references · 13

  1. 2017Neural Discrete Representation Learningcited by 4,650
  2. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  3. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,179
  4. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  5. 2016Unifying Count-Based Exploration and Intrinsic Motivationcited by 632
  6. 2016#Exploration: A Study of Count-Based Exploration for Deep Reinforcement Learningcited by 307
  7. 2017Count-Based Exploration with Neural Density Modelscited by 255
  8. 2018Count-Based Exploration with the Successor Representationcited by 112
  9. 2023Online Clustered Codebookcited by 55
  10. 2021Vector Quantized Models for Planningcited by 47
  11. 2021MADE: Exploration via Maximizing Deviation from Explored Regionscited by 40
  12. 2023HQ-VAE: Hierarchical Discrete Representation Learning with Variational Bayescited by 27
1 more
  1. 2024Mitigating Adversarial Perturbations for Deep Reinforcement Learning via Vector Quantizationcited by 4

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Policy-Based Trajectory Clustering in Offline Reinforcement Learning
  2. 2021Interpretable performance analysis towards offline reinforcement learning: A dataset perspective
  3. 2021False Correlation Reduction for Offline Reinforcement Learning
  4. 2021A Dataset Perspective on Offline Reinforcement Learning
  5. 2024On Sample-Efficient Offline Reinforcement Learning: Data Diversity, Posterior Sampling, and Beyond
  6. 2023Offline Reinforcement Learning with Imbalanced Datasets
  7. 2023Towards Robust Offline Reinforcement Learning under Diverse Data Corruption
  8. 2024CDSA: Conservative Denoising Score-based Algorithm for Offline Reinforcement Learning