mlsanityData to 7 Oct 2026

Neurosymbolic Grounding for Compositional World Models

arXiv 2310.12690 ↗First posted 19 Oct 2023cs.LG cs.AI stat.MLcited by 14

On the map: World models → Latent predictive world models

Abstract (arXiv)

We introduce Cosmos, a framework for object-centric world modeling that is designed for compositional generalization (CompGen), i.e., high performance on unseen input scenes obtained through the composition of known visual "atoms." The central insight behind Cosmos is the use of a novel form of neurosymbolic grounding. Specifically, the framework introduces two new tools: (i) neurosymbolic scene encodings, which represent each entity in a scene using a real vector computed using a neural encoder, as well as a vector of composable symbols describing attributes of the entity, and (ii) a neurosymbolic attention mechanism that binds these entities to learned rules of interaction. Cosmos is end-to-end differentiable; also, unlike traditional neurosymbolic methods that require representations to be manually mapped to symbols, it computes an entity's symbolic attributes using vision-language foundation models. Through an evaluation that considers two different forms of CompGen on an established blocks-pushing domain, we show that the framework establishes a new state-of-the-art for CompGen in world modeling. Artifacts are available at: https://trishullab.github.io/cosmos-web/

Key facts. Cited by 14 papers in the map (data through 7 Oct 2026). Placed in Latent predictive world models, one of 11 directions in World models. Builds on 40 of its 45 references that are in the map. First posted on arXiv 19 Oct 2023.

Builds on

40 of 45 references.

Closest ancestors · 2

Same direction, newest first

  1. 2022Learning Symmetric Embeddings for Equivariant World Models24 Apr
  2. 2019Contrastive Learning of Structured World Models27 Nov

Other references · 38

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  4. 2016Categorical Reparameterization with Gumbel-Softmaxcited by 2,708
  5. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,081
  6. 2018World Modelscited by 1,444
  7. 2023Mastering Diverse Domains through World Modelscited by 1,193
  8. 2016Group Equivariant Convolutional Networkscited by 1,154
  9. 2020Object-Centric Learning with Slot Attentioncited by 865
  10. 2023ViperGPT: Visual Inference via Python Execution for Reasoningcited by 591
  11. 2022Visual Programming: Compositional visual reasoning without trainingcited by 507
  12. 2015Neural Module Networkscited by 375
26 more
  1. 2018Neural-Symbolic VQA: Disentangling Reasoning from Vision and Language Understandingcited by 304
  2. 2016Steerable CNNscited by 298
  3. 2019Measuring Compositional Generalization: A Comprehensive Method on Realistic Datacited by 280
  4. 2020MDP Homomorphic Networks: Group Symmetries in Reinforcement Learningcited by 133
  5. 2019Spatial Broadcast Decoder: A Simple Architecture for Learning Disentangled Representations in VAEscited by 116
  6. 2019Entity Abstraction in Visual Model-Based Reinforcement Learningcited by 110
  7. 2016Learning to Compose Neural Networks for Question Answeringcited by 85
  8. 2021Neural Production Systems: Learning Rule-Governed Visual Dynamicscited by 75
  9. 2017Learning to Infer Graphics Programs from Hand-Drawn Imagescited by 73
  10. 2022Learning to Compose Soft Prompts for Compositional Zero-Shot Learningcited by 66
  11. 2020Plannable Approximations to MDP Homomorphisms: Equivariance under Actionscited by 61
  12. 2022Improving Object-centric Learning with Query Optimizationcited by 60
  13. 2023NS3D: Neuro-Symbolic Grounding of 3D Objects and Relationscited by 57
  14. 2019COBRA: Data-Efficient Model-Based RL through Unsupervised Object Discovery and Curiosity-Driven Explorationcited by 55
  15. 2021Systematic Evaluation of Causal Discovery in Visual Model Based Reinforcement Learningcited by 53
  16. 2022Object Representations as Fixed Points: Training Iterative Refinement Algorithms with Implicit Differentiationcited by 46
  17. 2018HOUDINI: Lifelong Learning as Program Synthesiscited by 38
  18. 2020Learning Differentiable Programs with Admissible Neural Heuristicscited by 30
  19. 2020Task Programming: Learning Data Efficient Behavior Representationscited by 27
  20. 2020Trajectory Prediction using Equivariant Continuous Convolutioncited by 26
  21. 2023Programmatically Grounded, Compositionally Generalizable Robotic Manipulationcited by 26
  22. 2022Toward Compositional Generalization in Object-Oriented World Modelingcited by 22
  23. 2022From Perception to Programs: Regularize, Overparameterize, and Amortizecited by 13
  24. 2019Program-Guided Image Manipulatorscited by 12
  25. 2021Unsupervised Learning of Neurosymbolic Encoderscited by 8
  26. 2023Neural Constraint Satisfaction: Hierarchical Abstraction for Combinatorial Generalization in Object Rearrangementcited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 8; 2026: 3.

14 papers.

Sort:

By area:

  1. 2024Compositional Generative Modeling: A Single Model is Not All You Needcited by 41
  2. 2025Synthesizing world models for bilevel planningcited by 13
  3. 2025Dyn-O: Building Structured World Models with Object-Centric Representationscited by 12
  4. 2025Learning Interactive World Model for Object-Centric Reinforcement Learningcited by 11
  5. 2025Pelican-VL 1.0: A Foundation Brain Model for Embodied Intelligencecited by 10
  6. 2025Compose Your Policies! Improving Diffusion-based or Flow-based Robot Policies via Test-time Distribution-level Compositioncited by 8
  7. 2026Neuro-Symbolic Synergy for World Modelingcited by 6
  8. 2025Dreamweaver: Learning Compositional World Models from Pixelscited by 4
  9. 2025Bridging VLMs and Embodied Intelligence with Deliberate Practice Policy Optimizationcited by 3
  10. 2024Aligning Generalisation Between Humans and Machinescited by 2
  11. 2025Discrete JEPA: Learning Discrete Token Representations without Reconstructioncited by 2
  12. 2024In-Context Learning Improves Compositional Understanding of Vision-Language Modelscited by 1
2 more
  1. 2026Grounding vs. Compositionality: On the Non-Complementarity of Reasoning in Neuro-Symbolic Systemscited by 0
  2. 2026The Design and Composition of Structural Causal Decision Processescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Imagine the Unseen World: A Benchmark for Systematic Generalization in Visual World Models
  2. 2025COGITAO: A Visual Reasoning Framework To Study Compositionality & Generalization
  3. 2025Bridging the gap to real-world language-grounded visual concept learning
  4. 2026Compositional Neuro-Symbolic Reasoning
  5. 2025Concept-RuleNet: Grounded Multi-Agent Neurosymbolic Reasoning in Vision Language Models
  6. 2022When and why vision-language models behave like bags-of-words, and what to do about it?
  7. 2026WorldString: Actionable World Representation
  8. 2023SNeL: A Structured Neuro-Symbolic Language for Entity-Based Multimodal Scene Understanding