# Neurosymbolic Grounding for Compositional World Models

arXiv: https://arxiv.org/abs/2310.12690 · HTML: https://mlsanity.com/paper/2310.12690/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Placed in Latent predictive world models, one of 11 directions in World models.
- Builds on 40 of its 45 references that are in the map.
- First posted on arXiv 19 Oct 2023.

## Placed in

- World models / [Latent predictive world models](https://mlsanity.com/wm/latent-predictive-world-models/)

## Builds on (most cited first, 30 of 40)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Categorical Reparameterization with Gumbel-Softmax](https://mlsanity.com/paper/1611.01144/) (2016)
- [Voyager: An Open-Ended Embodied Agent with Large Language Models](https://mlsanity.com/paper/2305.16291/) (2023)
- [World Models](https://mlsanity.com/paper/1803.10122/) (2018)
- [Mastering Diverse Domains through World Models](https://mlsanity.com/paper/2301.04104/) (2023)
- [Group Equivariant Convolutional Networks](https://mlsanity.com/paper/1602.07576/) (2016)
- [Object-Centric Learning with Slot Attention](https://mlsanity.com/paper/2006.15055/) (2020)
- [ViperGPT: Visual Inference via Python Execution for Reasoning](https://mlsanity.com/paper/2303.08128/) (2023)
- [Visual Programming: Compositional visual reasoning without training](https://mlsanity.com/paper/2211.11559/) (2022)
- [Neural Module Networks](https://mlsanity.com/paper/1511.02799/) (2015)
- [Steerable CNNs](https://mlsanity.com/paper/1612.08498/) (2016)
- [Neural-Symbolic VQA: Disentangling Reasoning from Vision and Language Understanding](https://mlsanity.com/paper/1810.02338/) (2018)
- [Measuring Compositional Generalization: A Comprehensive Method on Realistic Data](https://mlsanity.com/paper/1912.09713/) (2019)
- [Contrastive Learning of Structured World Models](https://mlsanity.com/paper/1911.12247/) (2019)
- [MDP Homomorphic Networks: Group Symmetries in Reinforcement Learning](https://mlsanity.com/paper/2006.16908/) (2020)
- [Spatial Broadcast Decoder: A Simple Architecture for Learning Disentangled Representations in VAEs](https://mlsanity.com/paper/1901.07017/) (2019)
- [Entity Abstraction in Visual Model-Based Reinforcement Learning](https://mlsanity.com/paper/1910.12827/) (2019)
- [Learning to Compose Neural Networks for Question Answering](https://mlsanity.com/paper/1601.01705/) (2016)
- [Neural Production Systems: Learning Rule-Governed Visual Dynamics](https://mlsanity.com/paper/2103.01937/) (2021)
- [Learning to Infer Graphics Programs from Hand-Drawn Images](https://mlsanity.com/paper/1707.09627/) (2017)
- [Learning to Compose Soft Prompts for Compositional Zero-Shot Learning](https://mlsanity.com/paper/2204.03574/) (2022)
- [Plannable Approximations to MDP Homomorphisms: Equivariance under Actions](https://mlsanity.com/paper/2002.11963/) (2020)
- [Improving Object-centric Learning with Query Optimization](https://mlsanity.com/paper/2210.08990/) (2022)
- [COBRA: Data-Efficient Model-Based RL through Unsupervised Object Discovery and Curiosity-Driven Exploration](https://mlsanity.com/paper/1905.09275/) (2019)
- [NS3D: Neuro-Symbolic Grounding of 3D Objects and Relations](https://mlsanity.com/paper/2303.13483/) (2023)
- [Learning Symmetric Embeddings for Equivariant World Models](https://mlsanity.com/paper/2204.11371/) (2022)
- [Systematic Evaluation of Causal Discovery in Visual Model Based Reinforcement Learning](https://mlsanity.com/paper/2107.00848/) (2021)
- [Object Representations as Fixed Points: Training Iterative Refinement Algorithms with Implicit Differentiation](https://mlsanity.com/paper/2207.00787/) (2022)

## Built on by (most cited first, 14 of 14)

- [Compositional Generative Modeling: A Single Model is Not All You Need](https://mlsanity.com/paper/2402.01103/) (2024)
- [Synthesizing world models for bilevel planning](https://mlsanity.com/paper/2503.20124/) (2025)
- [Dyn-O: Building Structured World Models with Object-Centric Representations](https://mlsanity.com/paper/2507.03298/) (2025)
- [Learning Interactive World Model for Object-Centric Reinforcement Learning](https://mlsanity.com/paper/2511.02225/) (2025)
- [Pelican-VL 1.0: A Foundation Brain Model for Embodied Intelligence](https://mlsanity.com/paper/2511.00108/) (2025)
- [Compose Your Policies! Improving Diffusion-based or Flow-based Robot Policies via Test-time Distribution-level Composition](https://mlsanity.com/paper/2510.01068/) (2025)
- [Neuro-Symbolic Synergy for World Modeling](https://mlsanity.com/paper/2602.10480/) (2026)
- [Dreamweaver: Learning Compositional World Models from Pixels](https://mlsanity.com/paper/2501.14174/) (2025)
- [Bridging VLMs and Embodied Intelligence with Deliberate Practice Policy Optimization](https://mlsanity.com/paper/2511.16602/) (2025)
- [Aligning Generalisation Between Humans and Machines](https://mlsanity.com/paper/2411.15626/) (2024)
- [Discrete JEPA: Learning Discrete Token Representations without Reconstruction](https://mlsanity.com/paper/2506.14373/) (2025)
- [In-Context Learning Improves Compositional Understanding of Vision-Language Models](https://mlsanity.com/paper/2407.15487/) (2024)
- [Grounding vs. Compositionality: On the Non-Complementarity of Reasoning in Neuro-Symbolic Systems](https://mlsanity.com/paper/2604.26521/) (2026)
- [The Design and Composition of Structural Causal Decision Processes](https://mlsanity.com/paper/2605.02681/) (2026)

## Abstract (arXiv)

We introduce Cosmos, a framework for object-centric world modeling that is designed for compositional generalization (CompGen), i.e., high performance on unseen input scenes obtained through the composition of known visual "atoms." The central insight behind Cosmos is the use of a novel form of neurosymbolic grounding. Specifically, the framework introduces two new tools: (i) neurosymbolic scene encodings, which represent each entity in a scene using a real vector computed using a neural encoder, as well as a vector of composable symbols describing attributes of the entity, and (ii) a neurosymbolic attention mechanism that binds these entities to learned rules of interaction. Cosmos is end-to-end differentiable; also, unlike traditional neurosymbolic methods that require representations to be manually mapped to symbols, it computes an entity's symbolic attributes using vision-language foundation models. Through an evaluation that considers two different forms of CompGen on an established blocks-pushing domain, we show that the framework establishes a new state-of-the-art for CompGen in world modeling. Artifacts are available at: https://trishullab.github.io/cosmos-web/