mlsanityData to 7 Oct 2026

Structured World Belief for Reinforcement Learning in POMDP

arXiv 2107.08577 ↗First posted 19 Jul 2021cs.LG cs.AIcited by 21

On the map: World models → Latent predictive world models

Abstract (arXiv)

Object-centric world models provide structured representation of the scene and can be an important backbone in reinforcement learning and planning. However, existing approaches suffer in partially-observable environments due to the lack of belief states. In this paper, we propose Structured World Belief, a model for learning and inference of object-centric belief states. Inferred by Sequential Monte Carlo (SMC), our belief states provide multiple object-centric scene hypotheses. To synergize the benefits of SMC particles with object representations, we also propose a new object-centric dynamics model that considers the inductive bias of object permanence. This enables tracking of object states even when they are invisible for a long time. To further facilitate object tracking in this regime, we allow our model to attend flexibly to any spatial location in the image which was restricted in previous models. In experiments, we show that object-centric belief provides a more accurate and robust performance for filtering and generation. Furthermore, we show the efficacy of structured world belief in improving the performance of reinforcement learning, planning and supervised reasoning.

Key facts. Cited by 21 papers in the map (data through 7 Oct 2026). Placed in Latent predictive world models, one of 11 directions in World models. Builds on 42 of its 58 references that are in the map. First posted on arXiv 19 Jul 2021.

Builds on

42 of 58 references.

Closest ancestors · 1

Same direction, newest first

  1. 2019SCALOR: Generative World Models with Scalable Object Representations6 Oct

Other references · 41

  1. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,218
  2. 2020Object-Centric Learning with Slot Attentioncited by 865
  3. 2016MOT16: A Benchmark for Multi-Object Trackingcited by 531
  4. 2015Deep Recurrent Q-Learning for Partially Observable MDPscited by 457
  5. 2019MONet: Unsupervised Scene Decomposition and Representationcited by 388
  6. 2019Multi-Object Representation Learning with Iterative Variational Inferencecited by 359
  7. 2020On the Binding Problem in Artificial Neural Networkscited by 230
  8. 2016Attend, Infer, Repeat: Fast Scene Understanding with Generative Modelscited by 209
  9. 2020SPACE: Unsupervised Object-Oriented Scene Representation via Spatial Attention and Decompositioncited by 190
  10. 2018Sequential Attend, Infer, Repeat: Generative Modelling of Moving Objectscited by 128
  11. 2017Neural Expectation Maximizationcited by 123
  12. 2018Deep Variational Reinforcement Learning for POMDPscited by 122
29 more
  1. 2018Unsupervised Learning of Object Landmarks through Conditional Image Generationcited by 113
  2. 2019Entity Abstraction in Visual Model-Based Reinforcement Learningcited by 110
  3. 2019Unsupervised Learning of Object Keypoints for Perception and Controlcited by 109
  4. 2020Bootstrap Latent-Predictive Representations for Multitask Reinforcement Learningcited by 101
  5. 2018Relational Neural Expectation Maximization: Unsupervised Discovery of Objects and their Interactionscited by 100
  6. 2017Variational Sequential Monte Carlocited by 93
  7. 2020Improving Generative Imagination in Object-Centric World Modelscited by 70
  8. 2019Shaping Belief States with Generative Environment Models for RLcited by 65
  9. 2017Auto-Encoding Sequential Monte Carlocited by 64
  10. 2020Generative Neurosymbolic Machinescited by 57
  11. 2019COBRA: Data-Efficient Model-Based RL through Unsupervised Object Discovery and Curiosity-Driven Explorationcited by 55
  12. 2017QMDP-Net: Deep Learning for Planning under Partial Observabilitycited by 51
  13. 2018Reasoning About Physical Interactions with Object-Oriented Prediction and Planningcited by 51
  14. 2017Deep Object-Centric Representations for Generalizable Robot Learningcited by 50
  15. 2018On Improving Deep Reinforcement Learning for POMDPscited by 49
  16. 2020Object Files and Schemata: Factorizing Declarative and Procedural Knowledge in Dynamical Systemscited by 48
  17. 2019Structured Object-Aware Physics Prediction for Video Modeling and Planningcited by 42
  18. 2019Exploiting Spatial Invariance for Scalable Unsupervised Object Trackingcited by 40
  19. 2020ROOTS: Object-Centric Representation and Rendering of 3D Scenescited by 40
  20. 2018Neural Predictive Belief Representationscited by 39
  21. 2019Particle Filter Recurrent Neural Networkscited by 36
  22. 2015State of the Art Control of Atari Games Using Shallow Reinforcement Learningcited by 23
  23. 2019Learning Belief Representations for Imitation Learning in POMDPscited by 23
  24. 2019DualSMC: Tunneling Differentiable Filtering and Planning under Continuous POMDPscited by 21
  25. 2018Unsupervised Video Object Segmentation for Deep Reinforcement Learningcited by 11
  26. 2018Variational Inference for Data-Efficient Model Learning in POMDPscited by 6
  27. 2020AlignNet: Unsupervised Entity Alignmentcited by 6
  28. 2020Investigating Simple Object Representations in Model-Free Deep Reinforcement Learningcited by 5
  29. 2020Relevance-Guided Modeling of Object Dynamics for Reinforcement Learningcited by 3

Built on by

Cited by year

Citing papers per year: 2021: 3; 2022: 7; 2023: 5; 2024: 4; 2025: 2.

21 papers.

Sort:

By area:

  1. 2022Simple Unsupervised Object-Centric Learning for Complex and Naturalistic Videoscited by 139
  2. 2023Object-Centric Slot Diffusioncited by 90
  3. 2023An Investigation into Pre-Training Object-Centric Representations for Reinforcement Learningcited by 49
  4. 2021Proximal Reinforcement Learning: Efficient Off-Policy Evaluation in Partially Observed Markov Decision Processescited by 40
  5. 2022Neural Systematic Bindercited by 37
  6. 2022Compositional Scene Representation Learning via Reconstruction: A Surveycited by 27
  7. 2022Unsupervised Multi-object Segmentation by Predicting Probable Motion Patternscited by 19
  8. 2024Exploring the Effectiveness of Object-Centric Representations in Visual Question Answering: Comparative Insights with Foundation Modelscited by 14
  9. 2022Latent State Marginalization as a Low-cost Approach for Improving Explorationcited by 13
  10. 2022A General Learning Framework for Open Ad Hoc Teamwork Using Graph-based Policy Learningcited by 12
  11. 2021Compositional Learning-based Planning for Vision POMDPscited by 8
  12. 2023DreamSmooth: Improving Model-based Reinforcement Learning via Reward Smoothingcited by 8
9 more
  1. 2023Seq2Seq Imitation Learning for Tactile Feedback-based Manipulationcited by 6
  2. 2024Compete and Compose: Learning Independent Mechanisms for Modular World Modelscited by 4
  3. 2025Object-Centric World Models for Causality-Aware Reinforcement Learningcited by 3
  4. 2025Sequential Monte Carlo for Policy Optimization in Continuous POMDPscited by 2
  5. 2021Dynamic programming with incomplete information to overcome navigational uncertainty in a nautical environmentcited by 1
  6. 2023Learning Successor Features with Distributed Hebbian Temporal Memorycited by 1
  7. 2024Solving Truly Massive Budgeted Monotonic POMDPs with Oracle-Guided Meta-Reinforcement Learningcited by 1
  8. 2022Multi-Step First: A Lightweight Deep Reinforcement Learning Strategy for Robust Continuous Control with Partial Observabilitycited by 0
  9. 2024Approximate Control for Continuous-Time POMDPscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Relational Object-Centric Actor-Critic
  2. 2026Object-Centric World Models Meet Monte Carlo Tree Search
  3. 2025When Object-Centric World Models Meet Policy Learning: From Pixels to Policies, and Where It Breaks
  4. 2026Slot-MPC: Goal-Conditioned Model Predictive Control with Object-Centric Representations
  5. 2025Learning Interactive World Model for Object-Centric Reinforcement Learning
  6. 2026Better Slots, Better Worlds: Representation Quality & Robustness in Object-Centric World Models
  7. 2025AXIOM: Learning to Play Games in Minutes with Expanding Object-Centric Models
  8. 2025Object-Centric World Models from Few-Shot Annotations for Sample-Efficient Reinforcement Learning