mlsanityData to 14 Oct 2026

Contrastive Learning of Structured World Models

arXiv 1911.12247 ↗First posted 27 Nov 2019stat.ML cs.AI cs.LGcited by 209

On the map: World models → Latent predictive world models

Abstract (arXiv)

A structured understanding of our world in terms of objects, relations, and hierarchies is an important component of human cognition. Learning such a structured world model from raw sensory data remains a challenge. As a step towards this goal, we introduce Contrastively-trained Structured World Models (C-SWMs). C-SWMs utilize a contrastive approach for representation learning in environments with compositional structure. We structure each state embedding as a set of object representations and their relations, modeled by a graph neural network. This allows objects to be discovered from raw pixel observations without direct supervision as part of the learning process. We evaluate C-SWMs on compositional environments involving multiple interacting objects that can be manipulated independently by an agent, simple Atari games, and a multi-object physics simulation. Our experiments demonstrate that C-SWMs can overcome limitations of models based on pixel reconstruction and outperform typical representatives of this model class in highly structured environments, while learning interpretable object-based representations.

Key facts. Cited by 209 papers in the map (data through 14 Oct 2026). Placed in Latent predictive world models, one of 11 directions in World models. Builds on 1 of its 73 references that are in the map. First posted on arXiv 27 Nov 2019.

Builds on

1 of 73 references.

Closest ancestors · 1

Same direction, newest first

  1. 2018Learning Latent Dynamics for Planning from Pixels12 Nov

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2020: 3; 2021: 41; 2022: 51; 2023: 29; 2024: 27; 2025: 19; 2026: 39.

209 papers.

Sort:

By area:

  1. 2021Graph Self-Supervised Learning: A Surveycited by 289
  2. 2021Conditional Object-Centric Learning from Videocited by 248
  3. 2022SAVi++: Towards End-to-End Object-Centric Learning from Real-World Videoscited by 166
  4. 2021Physion: Evaluating Physical Prediction from Vision in Humans and Machinescited by 153
  5. 2022SlotFormer: Unsupervised Visual Dynamics Simulation with Object-Centric Modelscited by 147
  6. 2022Simple Unsupervised Object-Centric Learning for Complex and Naturalistic Videoscited by 139
  7. 2021Self-supervised Learning on Graphs: Contrastive, Generative,or Predictivecited by 136
  8. 2024Web Agents with World Models: Learning and Leveraging Environment Dynamics in Web Navigationcited by 126
  9. 2020Offline Reinforcement Learning from Images with Latent Space Modelscited by 95
  10. 2023SlotDiffusion: Object-Centric Generative Modeling with Diffusion Modelscited by 85
  11. 2021Generalization and Robustness Implications in Object-Centric Learningcited by 81
  12. 2021Neural Production Systems: Learning Rule-Governed Visual Dynamicscited by 75
38 more
  1. 2021Greedy Hierarchical Variational Autoencoders for Large-Scale Video Predictioncited by 75
  2. 2024Learning and Leveraging World Models in Visual Representation Learningcited by 72
  3. 2022Self-Supervised Visual Representation Learning with Semantic Groupingcited by 70
  4. 2022Causal Dynamics Learning for Task-Independent State Abstractioncited by 65
  5. 2022Learning Multi-Object Dynamics with Compositional Neural Radiance Fieldscited by 63
  6. 2022TASKOGRAPHY: Evaluating robot task planning over large 3D scene graphscited by 59
  7. 2021Unsupervised Part Discovery from Contrastive Reconstructioncited by 57
  8. 2021Dynamic Visual Reasoning by Learning Differentiable Physics Models from Video and Languagecited by 55
  9. 2025LLaVE: Large Language and Vision Embedding Models with Hardness-Weighted Contrastive Learningcited by 55
  10. 2021Investigating the Role of Negatives in Contrastive Representation Learningcited by 54
  11. 2022Learning Symmetric Embeddings for Equivariant World Modelscited by 54
  12. 2021Systematic Evaluation of Causal Discovery in Visual Model Based Reinforcement Learningcited by 53
  13. 2020Deep Imitation Learning for Bimanual Robotic Manipulationcited by 50
  14. 2021CLINE: Contrastive Learning with Semantic Negative Examples for Natural Language Understandingcited by 50
  15. 2023An Investigation into Pre-Training Object-Centric Representations for Reinforcement Learningcited by 49
  16. 2023Self-supervised Object-Centric Learning for Videoscited by 49
  17. 2023Provably Learning Object-Centric Representationscited by 45
  18. 2024Dream to Manipulate: Compositional World Models Empowering Robot Imitation Learning with Imaginationcited by 45
  19. 2022MoCoDA: Model-based Counterfactual Data Augmentationcited by 44
  20. 2023Self-supervised learning of Split Invariant Equivariant representationscited by 44
  21. 2021PlayVirtual: Augmenting Cycle-Consistent Virtual Trajectories for Reinforcement Learningcited by 41
  22. 2023Provable Compositional Generalization for Object-Centric Learningcited by 41
  23. 2021Discrete-Valued Neural Communicationcited by 39
  24. 2022Deep Learning Methods for Abstract Visual Reasoning: A Survey on Raven's Progressive Matricescited by 37
  25. 2021Learning Knowledge Graph-based World Models of Textual Environmentscited by 31
  26. 2021Procedural Generalization by Planning with Self-Supervised World Modelscited by 31
  27. 2022Curious Exploration via Structured World Models Yields Zero-Shot Object Manipulationcited by 31
  28. 2024Artificial General Intelligence (AGI)-Native Wireless Systems: A Journey Beyond 6Gcited by 31
  29. 2024Temporally Consistent Object-Centric Learning by Contrasting Slotscited by 31
  30. 2021Which Mutual-Information Representation Learning Objectives are Sufficient for Control?cited by 29
  31. 2026Causal-JEPA: Learning World Models through Object-Level Latent Maskingcited by 29
  32. 2021Self-supervised Representation Learning with Relative Predictive Codingcited by 28
  33. 2022Non-Contrastive Learning Meets Language-Image Pre-Trainingcited by 25
  34. 2025Particle-Grid Neural Dynamics for Learning Deformable Object Models from RGB-D Videoscited by 25
  35. 2021CausalCity: Complex Simulations with Agency for Causal Discovery and Reasoningcited by 23
  36. 2022Policy Architectures for Compositional Generalization in Controlcited by 23
  37. 2022Symmetry Group Equivariant Architectures for Physicscited by 23
  38. 2022The Neural Process Family: Survey, Applications and Perspectivescited by 23

50 most cited of 209 All 209 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Dyn-O: Building Structured World Models with Object-Centric Representations
  2. 2026Platonic Representation Hypothesis on World Models
  3. 2026Learning Task-Sufficient World Models by Synergizing Agentic Exploration and Structured Modeling
  4. 2026WorldString: Actionable World Representation
  5. 2026Physical Object Understanding with a Physically Controllable World Model
  6. 2026What Do World Models Learn in RL? Probing Latent Representations in Learned Environment Simulators
  7. 2025Cloning Deterministic Worlds: The Critical Role of Latent Geometry in Long-Horizon World Models
  8. 2023ReCoRe: Regularized Contrastive Representation Learning of World Model