mlsanityData to 7 Oct 2026

Shaping Belief States with Generative Environment Models for RL

arXiv 1906.09237 ↗First posted 21 Jun 2019cs.LG cs.AI stat.MLcited by 65

Not on the map. Why?

Abstract (arXiv)

When agents interact with a complex environment, they must form and maintain beliefs about the relevant aspects of that environment. We propose a way to efficiently train expressive generative models in complex environments. We show that a predictive algorithm with an expressive generative model can form stable belief-states in visually rich and dynamic 3D environments. More precisely, we show that the learned representation captures the layout of the environment as well as the position and orientation of the agent. Our experiments show that the model substantially improves data-efficiency on a number of reinforcement learning (RL) tasks compared with strong model-free baseline agents. We find that predicting multiple steps into the future (overshooting), in combination with an expressive generative model, is critical for stable representations to emerge. In practice, using expressive generative models in RL is computationally expensive and we propose a scheme to reduce this computational burden, allowing us to build agents that are competitive with model-free baselines.

Key facts. Cited by 65 papers in the map (data through 7 Oct 2026). First posted on arXiv 21 Jun 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 21; 2022: 19; 2023: 10; 2024: 7; 2025: 5; 2026: 3.

65 papers.

Sort:

By area:

  1. 2021Towards Causal Representation Learningcited by 207
  2. 2021Recurrent Model-Free RL Can Be a Strong Baseline for Many POMDPscited by 115
  3. 2022Deep Hierarchical Planning from Pixelscited by 96
  4. 2023DREAMWALKER: Mental Planning for Continuous Vision-Language Navigationcited by 82
  5. 2021DreamerPro: Reconstruction-Free Model-Based Reinforcement Learning with Prototypical Representationscited by 73
  6. 2022BYOL-Explore: Exploration by Bootstrapped Predictioncited by 71
  7. 2022CACTI: A Framework for Scalable Multi-Task Multi-Scene Visual Imitation Learningcited by 70
  8. 2024Scaling Instructable Agents Across Many Simulated Worldscited by 67
  9. 2021Muesli: Combining Improvements in Policy Optimizationcited by 53
  10. 2022Optimizing Sequential Experimental Design with Deep Reinforcement Learningcited by 45
  11. 2022Evaluating Long-Term Memory in 3D Mazescited by 41
  12. 2021Tell me why! Explanations support learning relational and causal structurecited by 36
38 more
  1. 2021Procedural Generalization by Planning with Self-Supervised World Modelscited by 30
  2. 2021Attention Approximates Sparse Distributed Memorycited by 29
  3. 2022INFOrmation Prioritization through EmPOWERment in Visual Model-Based RLcited by 27
  4. 2022Towards Versatile Embodied Navigationcited by 26
  5. 2021Value Function Spaces: Skill-Centric State Abstractions for Long-Horizon Reasoningcited by 25
  6. 2021Differentiable Spatial Planning using Transformerscited by 25
  7. 2021Structured World Belief for Reinforcement Learning in POMDPcited by 21
  8. 2023Selective Visual Representations Improve Convergence and Generalization for Embodied AIcited by 19
  9. 2021Neural Recursive Belief States in Multi-Agent Reinforcement Learningcited by 18
  10. 2022Unsupervised Representation Learning in Deep Reinforcement Learning: A Reviewcited by 18
  11. 2022The Benefits of Model-Based Generalization in Reinforcement Learningcited by 17
  12. 2022Flow-based Recurrent Belief State Learning for POMDPscited by 16
  13. 2022Latent State Marginalization as a Low-cost Approach for Improving Explorationcited by 13
  14. 2021Environment Predictive Coding for Embodied Agentscited by 12
  15. 2021Temporally Abstract Partial Modelscited by 11
  16. 2023Informed POMDP: Leveraging Additional Information in Model-Based RLcited by 11
  17. 2023Would I have gotten that reward? Long-term credit assignment by counterfactual contribution analysiscited by 10
  18. 2023Provable Representation with Efficient Planning for Partial Observable Reinforcement Learningcited by 10
  19. 2024Do Transformer World Models Give Better Policy Gradients?cited by 10
  20. 2023The Wasserstein Believer: Learning Belief Updates for Partially Observable Environments through Reliable Latent Space Modelscited by 9
  21. 2025Beyond Generative AI: World Models for Clinical Prediction, Counterfactuals, and Planningcited by 9
  22. 2025Zero-Shot Adaptation of Behavioral Foundation Models to Unseen Dynamicscited by 8
  23. 2026Agent-BRACE: Decoupling Beliefs from Actions in Long-Horizon Tasks via Verbalized State Uncertaintycited by 8
  24. 2021Self-Consistent Models and Valuescited by 7
  25. 2023Graph schemas as abstractions for transfer learning, inference, and planningcited by 7
  26. 2021Learning State Representations from Random Deep Action-conditional Predictionscited by 6
  27. 2021Model-Value Inconsistency as a Signal for Epistemic Uncertaintycited by 6
  28. 2021Towards Disturbance-Free Visual Mobile Manipulationcited by 6
  29. 2022Safe Deep RL in 3D Environments using Human Feedbackcited by 5
  30. 2024Spatially-Aware Transformer for Embodied Agentscited by 5
  31. 2025A Theoretical Justification for Asymmetric Actor-Critic Algorithmscited by 5
  32. 2021A Whole Brain Probabilistic Generative Model: Toward Realizing Cognitive Architectures for Developmental Robotscited by 4
  33. 2021How Are Learned Perception-Based Controllers Impacted by the Limits of Robust Control?cited by 4
  34. 2022Curiosity in Hindsight: Intrinsic Exploration in Stochastic Environmentscited by 4
  35. 2022Tracking and Planning with Spatial World Modelscited by 3
  36. 2022Deterministic training of generative autoencoders using invertible layerscited by 3
  37. 2022A General Purpose Supervisory Signal for Embodied Agentscited by 3
  38. 2024An Empirical Study on the Power of Future Prediction in Partially Observable Environmentscited by 3

50 most cited of 65 All 65 (JSON)