mlsanityData to 7 Oct 2026

Vector Quantized Models for Planning

arXiv 2106.04615 ↗First posted 8 Jun 2021cs.LG cs.AI stat.MLcited by 47

Not on the map. Why?

Abstract (arXiv)

Recent developments in the field of model-based RL have proven successful in a range of environments, especially ones where planning is essential. However, such successes have been limited to deterministic fully-observed environments. We present a new approach that handles stochastic and partially-observable environments. Our key insight is to use discrete autoencoders to capture the multiple possible effects of an action in a stochastic environment. We use a stochastic variant of Monte Carlo tree search to plan over both the agent's actions and the discrete latent variables representing the environment's response. Our approach significantly outperforms an offline version of MuZero on a stochastic interpretation of chess where the opponent is considered part of the environment. We also show that our approach scales to DeepMind Lab, a first-person 3D environment with large visual observations and partial observability.

Key facts. Cited by 47 papers in the map (data through 7 Oct 2026). Builds on 38 of its 64 references that are in the map. First posted on arXiv 8 Jun 2021.

Builds on

38 of 64 references.

References · 38

  1. 2017Attention Is All You Needcited by 59,932
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2017Neural Discrete Representation Learningcited by 4,650
  5. 2013Auto-Encoding Variational Bayescited by 3,980
  6. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,218
  7. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,031
  8. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,598
  9. 2019Generating Diverse High-Fidelity Images with VQ-VAE-2cited by 1,376
  10. 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,327
  11. 2018Learning Latent Dynamics for Planning from Pixelscited by 1,221
  12. 2019Dota 2 with Large Scale Deep Reinforcement Learningcited by 1,084
26 more
  1. 2020Mastering Atari with Discrete World Modelscited by 990
  2. 2018Recurrent World Models Facilitate Policy Evolutioncited by 761
  3. 2018Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Modelscited by 748
  4. 2017Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithmcited by 723
  5. 2018Learning Dexterous In-Hand Manipulationcited by 643
  6. 2020MOPO: Model-based Offline Policy Optimizationcited by 630
  7. 2020MOReL : Model-Based Offline Reinforcement Learningcited by 582
  8. 2019Model-Based Reinforcement Learning for Ataricited by 495
  9. 2020dm_control: Software and Tasks for Continuous Controlcited by 368
  10. 2017Revisiting the Arcade Learning Environment: Evaluation Protocols and Open Problems for General Agentscited by 267
  11. 2017Thinking Fast and Slow with Deep Learning and Tree Searchcited by 245
  12. 2017DeepStack: Expert-Level Artificial Intelligence in No-Limit Pokercited by 228
  13. 2018Model-Ensemble Trust-Region Policy Optimizationcited by 214
  14. 2015Action-Conditional Video Prediction using Deep Networks in Atari Gamescited by 171
  15. 2016Deep Reinforcement Learning from Self-Play in Imperfect-Information Gamescited by 153
  16. 2020Model-Based Offline Planningcited by 126
  17. 2019When to use parametric models in reinforcement learning?cited by 118
  18. 2017Value Prediction Networkcited by 117
  19. 2017Building machines that adapt and compute like brainscited by 99
  20. 2018Taming VAEscited by 85
  21. 2017Recurrent Environment Simulatorscited by 68
  22. 2020RL Unplugged: A Suite of Benchmarks for Offline Reinforcement Learningcited by 57
  23. 2016Learning and Policy Search in Stochastic Dynamical Systems with Bayesian Neural Networkscited by 49
  24. 2018Learning and Querying Fast Generative Models for Reinforcement Learningcited by 48
  25. 2020Causally Correct Partial Models for Reinforcement Learningcited by 24
  26. 2017Learning Multimodal Transition Dynamics for Model-Based Reinforcement Learningcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 2; 2022: 6; 2023: 17; 2024: 12; 2025: 6; 2026: 4.

47 papers.

Sort:

By area:

  1. 2022Planning with Diffusion for Flexible Behavior Synthesiscited by 1,023
  2. 2022Transformers are Sample-Efficient World Modelscited by 317
  3. 2023Foundation Models for Decision Making: Problems, Methods, and Opportunitiescited by 160
  4. 2024CycleResearcher: Improving Automated Research via Automated Reviewcited by 142
  5. 2021Gradients are Not All You Needcited by 82
  6. 2023Straightening Out the Straight-Through Estimator: Overcoming Optimization Challenges in Vector Quantized Networkscited by 80
  7. 2023SafeDiffuser: Safe Planning with Diffusion Probabilistic Modelscited by 69
  8. 2022Efficient Planning in a Compact Latent Action Spacecited by 57
  9. 2023A Survey on Transformers in Reinforcement Learningcited by 50
  10. 2023PlayFusion: Skill Acquisition via Diffusion from Language-Annotated Playcited by 48
  11. 2022On Transforming Reinforcement Learning by Transformer: The Development Trajectorycited by 35
  12. 2024Efficient World Models with Context-Aware Tokenizationcited by 33
35 more
  1. 2021Procedural Generalization by Planning with Self-Supervised World Modelscited by 30
  2. 2022The Free Energy Principle for Perception and Action: A Deep Learning Perspectivecited by 24
  3. 2022Addressing Optimism Bias in Sequence Modeling for Reinforcement Learningcited by 22
  4. 2023LightZero: A Unified Benchmark for Monte Carlo Tree Search in General Sequential Decision Scenarioscited by 20
  5. 2023Hierarchical Imitation Learning with Vector Quantized Modelscited by 15
  6. 2023BetaZero: Belief-State Planning for Long-Horizon POMDPs using Learned Approximationscited by 13
  7. 2023Efficient Planning with Latent Diffusioncited by 11
  8. 2024Fine-Grained Causal Dynamics Learning with Quantization for Improving Robustness in Reinforcement Learningcited by 10
  9. 2024Learning from Random Demonstrations: Offline Reinforcement Learning with Importance-Sampled Diffusion Modelscited by 8
  10. 2023Skill Decision Transformercited by 7
  11. 2023Hybrid Search for Efficient Planning with Completeness Guaranteescited by 6
  12. 2024Efficient Monte Carlo Tree Search via On-the-Fly State-Conditioned Action Abstractioncited by 6
  13. 2024The Impact of Quantization and Pruning on Deep Reinforcement Learning Modelscited by 6
  14. 2024Planning Transformer: Long-Horizon Offline Reinforcement Learning with Planning Tokenscited by 4
  15. 2024Mitigating Adversarial Perturbations for Deep Reinforcement Learning via Vector Quantizationcited by 4
  16. 2024Few-Shot Task Learning through Inverse Generative Modelingcited by 4
  17. 2025Scalable Decision-Making in Stochastic Environments through Learned Temporal Abstractioncited by 4
  18. 2023GenCO: Generating Diverse Designs with Combinatorial Constraintscited by 3
  19. 2025AUPO -- Abstracted Until Proven Otherwise: A Reward Distribution Based Abstraction Algorithmcited by 3
  20. 2023Streamlining Multimodal Data Fusion in Wireless Communication and Sensor Networkscited by 2
  21. 2025Investigating Intra-Abstraction Policies For Non-exact Abstraction Algorithmscited by 2
  22. 2025Learning Human-Like RL Agents Through Trajectory Optimization With Action Quantizationcited by 2
  23. 2023Deep Generative Models for Decision-Making and Controlcited by 1
  24. 2023Learning Disentangled Discrete Representationscited by 1
  25. 2024Transformer Based Planning in the Observation Space with Applications to Trick Taking Card Gamescited by 1
  26. 2024Machine Learning in Short-Reach Optical Systems: A Comprehensive Surveycited by 1
  27. 2024Enhancing Interpretability in Deep Reinforcement Learning through Semantic Clusteringcited by 1
  28. 2026Revisiting Regularized Policy Optimization for Stable and Efficient Reinforcement Learning in Two-Player Gamescited by 1
  29. 2026Motion Planning in Compressed Representation Spacescited by 1
  30. 2023Self-Organising Neural Discrete Representation Learning `a la Kohonencited by 0
  31. 2023Know your Enemy: Investigating Monte-Carlo Tree Search with Opponent Models in Pommermancited by 0
  32. 2025Task-Driven Discrete Representation Learningcited by 0
  33. 2025Discovering State Equivalences in UCT Search Trees By Action Pruningcited by 0
  34. 2026STO-RL: Offline RL under Sparse Rewards via LLM-Guided Subgoal Temporal Ordercited by 0
  35. 2026Efficient Anti-exploration via VQVAE and Fuzzy Clustering in Offline Reinforcement Learningcited by 0