# Discovering Predictive Relational Object Symbols with Symbolic Attentive Layers

arXiv: https://arxiv.org/abs/2309.00889 · HTML: https://mlsanity.com/paper/2309.00889/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 40 references that are in the map.
- First posted on arXiv 2 Sep 2023.

## Builds on (most cited first, 20 of 20)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Categorical Reparameterization with Gumbel-Softmax](https://mlsanity.com/paper/1611.01144/) (2016)
- [The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables](https://mlsanity.com/paper/1611.00712/) (2016)
- [Prioritized Experience Replay](https://mlsanity.com/paper/1511.05952/) (2015)
- [Classical Planning in Deep Latent Space: Bridging the Subsymbolic-Symbolic Boundary](https://mlsanity.com/paper/1705.00154/) (2017)
- [Learning Symbolic Operators for Task and Motion Planning](https://mlsanity.com/paper/2103.00589/) (2021)
- [Learning Neuro-Symbolic Skills for Bilevel Planning](https://mlsanity.com/paper/2206.10680/) (2022)
- [Predicate Invention for Bilevel Planning](https://mlsanity.com/paper/2203.09634/) (2022)
- [Learning Neuro-Symbolic Relational Transition Models for Bilevel Planning](https://mlsanity.com/paper/2105.14074/) (2021)
- [Symbol Emergence in Cognitive Developmental Systems: a Survey](https://mlsanity.com/paper/1801.08829/) (2018)
- [Understanding the Power and Limitations of Teaching with Imperfect Knowledge](https://mlsanity.com/paper/2003.09712/) (2020)
- [Classical Planning in Deep Latent Space](https://mlsanity.com/paper/2107.00110/) (2021)
- [Planning for Multi-Object Manipulation with Graph Neural Network Relational Classifiers](https://mlsanity.com/paper/2209.11943/) (2022)
- [Learning Portable Representations for High-Level Planning](https://mlsanity.com/paper/1905.12006/) (2019)
- [Embodied Active Learning of Relational State Abstractions for Bilevel Planning](https://mlsanity.com/paper/2303.04912/) (2023)
- [Belief Regulated Dual Propagation Nets for Learning Action Effects on Groups of Articulated Objects](https://mlsanity.com/paper/1909.03785/) (2019)
- [Object and Relation Centric Representations for Push Effect Prediction](https://mlsanity.com/paper/2102.02100/) (2021)
- [Learning Temporally Extended Skills in Continuous Domains as Symbolic Actions for Planning](https://mlsanity.com/paper/2207.05018/) (2022)
- [High-level Features for Resource Economy and Fast Learning in Skill Transfer](https://mlsanity.com/paper/2106.10354/) (2021)

## Built on by (most cited first, 3 of 3)

- [Neuro-Symbolic Imitation Learning: Discovering Symbolic Abstractions for Skill Learning](https://mlsanity.com/paper/2503.21406/) (2025)
- [Neuro-Symbolic Skill Discovery for Conditional Multi-Level Planning](https://mlsanity.com/paper/2410.10045/) (2024)
- [Macro-Operator Generation and Predicate Selection for TAMP Operator Learning](https://mlsanity.com/paper/2608.23629/) (2026)

## Abstract (arXiv)

In this paper, we propose and realize a new deep learning architecture for discovering symbolic representations for objects and their relations based on the self-supervised continuous interaction of a manipulator robot with multiple objects on a tabletop environment. The key feature of the model is that it can handle a changing number number of objects naturally and map the object-object relations into symbolic domain explicitly. In the model, we employ a self-attention layer that computes discrete attention weights from object features, which are treated as relational symbols between objects. These relational symbols are then used to aggregate the learned object symbols and predict the effects of executed actions on each object. The result is a pipeline that allows the formation of object symbols and relational symbols from a dataset of object features, actions, and effects in an end-to-end manner. We compare the performance of our proposed architecture with state-of-the-art symbol discovery methods in a simulated tabletop environment where the robot needs to discover symbols related to the relative positions of objects to predict the observed effect successfully. Our experiments show that the proposed architecture performs better than other baselines in effect prediction while forming not only object symbols but also relational symbols. Furthermore, we analyze the learned symbols and relational patterns between objects to learn about how the model interprets the environment. Our analysis shows that the learned symbols relate to the relative positions of objects, object types, and their horizontal alignment on the table, which reflect the regularities in the environment.