# Object-centric proto-symbolic behavioural reasoning from pixels arXiv: https://arxiv.org/abs/2411.17438 · HTML: https://mlsanity.com/paper/2411.17438/ ## Key facts - Cited by 2 papers in the map (data through 14 Oct 2026). - Builds on 46 of its 62 references that are in the map. - First posted on arXiv 26 Nov 2024. ## Builds on (most cited first, 30 of 46) - [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018) - [Categorical Reparameterization with Gumbel-Softmax](https://mlsanity.com/paper/1611.01144/) (2016) - [Do As I Can, Not As I Say: Grounding Language in Robotic Affordances](https://mlsanity.com/paper/2204.01691/) (2022) - [PaLM-E: An Embodied Multimodal Language Model](https://mlsanity.com/paper/2303.03378/) (2023) - [Object-Centric Learning with Slot Attention](https://mlsanity.com/paper/2006.15055/) (2020) - [Interaction Networks for Learning about Objects, Relations and Physics](https://mlsanity.com/paper/1612.00222/) (2016) - [Multi-Object Representation Learning with Iterative Variational Inference](https://mlsanity.com/paper/1903.00450/) (2019) - [Inductive Biases for Deep Learning of Higher-Level Cognition](https://mlsanity.com/paper/2011.15091/) (2020) - [Conditional Object-Centric Learning from Video](https://mlsanity.com/paper/2111.12594/) (2021) - [On the Binding Problem in Artificial Neural Networks](https://mlsanity.com/paper/2012.05208/) (2020) - [Attend, Infer, Repeat: Fast Scene Understanding with Generative Models](https://mlsanity.com/paper/1603.08575/) (2016) - [PredRNN: A Recurrent Neural Network for Spatiotemporal Predictive Learning](https://mlsanity.com/paper/2103.09504/) (2021) - [SAVi++: Towards End-to-End Object-Centric Learning from Real-World Videos](https://mlsanity.com/paper/2206.07764/) (2022) - [RT-Trajectory: Robotic Task Generalization via Hindsight Trajectory Sketches](https://mlsanity.com/paper/2311.01977/) (2023) - [Entity Abstraction in Visual Model-Based Reinforcement Learning](https://mlsanity.com/paper/1910.12827/) (2019) - [SCALOR: Generative World Models with Scalable Object Representations](https://mlsanity.com/paper/1910.02384/) (2019) - [Unsupervised Discovery of Object Radiance Fields](https://mlsanity.com/paper/2107.07905/) (2021) - [Visual Interaction Networks](https://mlsanity.com/paper/1706.01433/) (2017) - [Iterative Amortized Inference](https://mlsanity.com/paper/1807.09356/) (2018) - [Object-Centric Slot Diffusion](https://mlsanity.com/paper/2303.10834/) (2023) - [LLMs and the Abstraction and Reasoning Corpus: Successes, Failures, and the Importance of Object-based Representations](https://mlsanity.com/paper/2305.18354/) (2023) - [Deep active inference agents using Monte-Carlo methods](https://mlsanity.com/paper/2006.04176/) (2020) - [Learning and Leveraging World Models in Visual Representation Learning](https://mlsanity.com/paper/2403.00504/) (2024) - [Propagation Networks for Model-Based Control Under Partial Observation](https://mlsanity.com/paper/1809.11169/) (2018) - [Learning Multi-Object Dynamics with Compositional Neural Radiance Fields](https://mlsanity.com/paper/2202.11855/) (2022) - [Active Inference in Robotics and Artificial Agents: Survey and Challenges](https://mlsanity.com/paper/2112.01871/) (2021) - [Self-supervised Visual Reinforcement Learning with Object-centric Representations](https://mlsanity.com/paper/2011.14381/) (2020) - [RT-Sketch: Goal-Conditioned Imitation Learning from Hand-Drawn Sketches](https://mlsanity.com/paper/2403.02709/) (2024) - [Whence the Expected Free Energy?](https://mlsanity.com/paper/2004.08128/) (2020) - [World Models and Predictive Coding for Cognitive and Developmental Robotics: Frontiers and Challenges](https://mlsanity.com/paper/2301.05832/) (2023) ## Built on by (most cited first, 2 of 2) - [Cognitively-Inspired Emergent Communication via Knowledge Graphs for Assisting the Visually Impaired](https://mlsanity.com/paper/2505.22087/) (2025) - [Mechanistic Foundations of Goal-Directed Control](https://mlsanity.com/paper/2603.15248/) (2026) ## Abstract (arXiv) Autonomous intelligent agents must bridge computational challenges at disparate levels of abstraction, from the low-level spaces of sensory input and motor commands to the high-level domain of abstract reasoning and planning. A key question in designing such agents is how best to instantiate the representational space that will interface between these two levels -- ideally without requiring supervision in the form of expensive data annotations. These objectives can be efficiently achieved by representing the world in terms of objects (grounded in perception and action). In this work, we present a novel, brain-inspired, deep-learning architecture that learns from pixels to interpret, control, and reason about its environment, using object-centric representations. We show the utility of our approach through tasks in synthetic environments that require a combination of (high-level) logical reasoning and (low-level) continuous control. Results show that the agent can learn emergent conditional behavioural reasoning, such as (A → B) land (neg A → C), as well as logical composition (A → B) land (A → C) vdash A → (B land C) and XOR operations, and successfully controls its environment to satisfy objectives deduced from these logical rules. The agent can adapt online to unexpected changes in its environment and is robust to mild violations of its world model, thanks to dynamic internal desired goal generation. While the present results are limited to synthetic settings (2D and 3D activated versions of dSprites), which fall short of real-world levels of complexity, the proposed architecture shows how to manipulate grounded object representations, as a key inductive bias for unsupervised learning, to enable behavioral reasoning.