mlsanityData to 14 Oct 2026

Deep Reinforcement Learning with Vector Quantized Encoding

arXiv 2211.06733 ↗First posted 12 Nov 2022cs.LGcited by 2

Not on the map. Why?

Abstract (arXiv)

Human decision-making often involves combining similar states into categories and reasoning at the level of the categories rather than the actual states. Guided by this intuition, we propose a novel method for clustering state features in deep reinforcement learning (RL) methods to improve their interpretability. Specifically, we propose a plug-and-play framework termed vector quantized reinforcement learning (VQ-RL) that extends classic RL pipelines with an auxiliary classification task based on vector quantized (VQ) encoding and aligns with policy training. The VQ encoding method categorizes features with similar semantics into clusters and results in tighter clusters with better separation compared to classic deep RL methods, thus enabling neural models to learn similarities and differences between states better. Furthermore, we introduce two regularization methods to help increase the separation between clusters and avoid the risks associated with VQ training. In simulations, we demonstrate that VQ-RL improves interpretability and investigate its impact on robustness and generalization of deep RL.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 34 of its 43 references that are in the map. First posted on arXiv 12 Nov 2022.

Builds on

34 of 43 references.

References · 34

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  3. 2017Neural Discrete Representation Learningcited by 4,712
  4. 2013Auto-Encoding Variational Bayescited by 4,053
  5. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,258
  6. 2019Generating Diverse High-Fidelity Images with VQ-VAE-2cited by 1,387
  7. 2015End-to-End Training of Deep Visuomotor Policiescited by 875
  8. 2017A Distributional Perspective on Reinforcement Learningcited by 765
  9. 2021VideoGPT: Video Generation using VQ-VAE and Transformerscited by 658
  10. 2019Model-Based Reinforcement Learning for Ataricited by 502
  11. 2017A Brief Survey of Deep Reinforcement Learningcited by 483
  12. 2019Leveraging Procedural Generation to Benchmark Reinforcement Learningcited by 483
22 more
  1. 2016Reinforcement Learning with Unsupervised Auxiliary Taskscited by 404
  2. 2018Quantifying Generalization in Reinforcement Learningcited by 352
  3. 2020Decoupling Representation Learning from Reinforcement Learningcited by 269
  4. 2018Human-level performance in first-person multiplayer games with population-based deep reinforcement learningcited by 264
  5. 2019Stochastic Latent Actor-Critic: Deep Reinforcement Learning with a Latent Variable Modelcited by 231
  6. 2018Programmatically Interpretable Reinforcement Learningcited by 193
  7. 2019Action Robust Reinforcement Learning and Applications in Continuous Controlcited by 174
  8. 2018Assessing Generalization in Deep Reinforcement Learningcited by 133
  9. 2019Network Randomization: A Simple Technique for Generalization in Deep Reinforcement Learningcited by 104
  10. 2020Bootstrap Latent-Predictive Representations for Multitask Reinforcement Learningcited by 102
  11. 2019Generalization in Reinforcement Learning with Selective Noise Injection and Information Bottleneckcited by 101
  12. 2018A Dissection of Overfitting and Generalization in Continuous Reinforcement Learningcited by 87
  13. 2019Towards Interpretable Reinforcement Learning Using Attention Augmented Agentscited by 82
  14. 2018Reinforcement Learning with Perturbed Rewardscited by 79
  15. 2018SDRL: Interpretable and Data-efficient Deep Reinforcement Learning Leveraging Symbolic Planningcited by 78
  16. 2016Graying the black box: Understanding DQNscited by 68
  17. 2019Low Bit-Rate Speech Coding with VQ-VAE and a WaveNet Decodercited by 49
  18. 2019Certified Adversarial Robustness for Deep Reinforcement Learningcited by 47
  19. 2018Combined Reinforcement Learning via Abstract Representationscited by 37
  20. 2020Transformer VQ-VAE for Unsupervised Unit Discovery and Speech Synthesis: ZeroSpeech 2020 Challengecited by 21
  21. 2018An Atari Model Zoo for Analyzing, Visualizing, and Comparing Deep Reinforcement Learning Agentscited by 15
  22. 2020Smaller World Models for Reinforcement Learningcited by 4

Built on by

Cited by year

Citing papers per year: 2024: 2.

2 papers.

  1. 2024Learning to Play Atari in a World of Tokenscited by 6
  2. 2024The Interpretability of Codebooks in Model-Based Reinforcement Learning is Limitedcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Enhancing Interpretability in Deep Reinforcement Learning through Semantic Clustering
  2. 2024iQRL -- Implicitly Quantized Representations for Sample-efficient Reinforcement Learning
  3. 2025A Survey on Explainable Deep Reinforcement Learning
  4. 2025Learning Distinguishable Representations in Deep Q-Networks for Linear Transfer
  5. 2026QuRL: Efficient Reinforcement Learning with Quantized Rollout
  6. 2024Mitigating Adversarial Perturbations for Deep Reinforcement Learning via Vector Quantization
  7. 2026Critic-Driven Voronoi-Quantization for Distilling Deep RL Policies to Explainable Models
  8. 2025A Survey of State Representation Learning for Deep Reinforcement Learning