# Binding Actions to Objects in World Models

arXiv: https://arxiv.org/abs/2204.13022 · HTML: https://mlsanity.com/paper/2204.13022/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Placed in Embodied and robot world models, one of 11 directions in World models.
- Builds on 13 of its 21 references that are in the map.
- First posted on arXiv 27 Apr 2022.

## Placed in

- World models / [Embodied and robot world models](https://mlsanity.com/wm/embodied-and-robot-world-models/)

## Builds on (most cited first, 13 of 13)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Relational inductive biases, deep learning, and graph networks](https://mlsanity.com/paper/1806.01261/) (2018)
- [On the Binding Problem in Artificial Neural Networks](https://mlsanity.com/paper/2012.05208/) (2020)
- [MineRL: A Large-Scale Dataset of Minecraft Demonstrations](https://mlsanity.com/paper/1907.13440/) (2019)
- [Contrastive Learning of Structured World Models](https://mlsanity.com/paper/1911.12247/) (2019)
- [Entity Abstraction in Visual Model-Based Reinforcement Learning](https://mlsanity.com/paper/1910.12827/) (2019)
- [Neural Production Systems: Learning Rule-Governed Visual Dynamics](https://mlsanity.com/paper/2103.01937/) (2021)
- [Towards Practical Multi-Object Manipulation using Relational Reinforcement Learning](https://mlsanity.com/paper/1912.11032/) (2019)
- [Reasoning About Physical Interactions with Object-Oriented Prediction and Planning](https://mlsanity.com/paper/1812.10972/) (2018)
- [Structured agents for physical construction](https://mlsanity.com/paper/1904.03177/) (2019)
- [Action Recognition using Visual Attention](https://mlsanity.com/paper/1511.04119/) (2015)
- [Factored World Models for Zero-Shot Generalization in Robotic Manipulation](https://mlsanity.com/paper/2202.05333/) (2022)
- [The Impact of Negative Sampling on Contrastive Structured World Models](https://mlsanity.com/paper/2107.11676/) (2021)

## Built on by (most cited first, 10 of 10)

- [Bridging the Gap to Real-World Object-Centric Learning](https://mlsanity.com/paper/2209.14860/) (2022)
- [Toward Compositional Generalization in Object-Oriented World Modeling](https://mlsanity.com/paper/2204.13661/) (2022)
- [Structure in Deep Reinforcement Learning: A Survey and Open Problems](https://mlsanity.com/paper/2306.16021/) (2023)
- [Learning Dynamic Attribute-factored World Models for Efficient Multi-object Reinforcement Learning](https://mlsanity.com/paper/2307.09205/) (2023)
- [GLASS: Guided Latent Slot Diffusion for Object-Centric Learning](https://mlsanity.com/paper/2407.17929/) (2024)
- [Object-Centric Temporal Consistency via Conditional Autoregressive Inductive Biases](https://mlsanity.com/paper/2410.15728/) (2024)
- [Relational Object-Centric Actor-Critic](https://mlsanity.com/paper/2310.17178/) (2023)
- [Learning Actionable World Models for Industrial Process Control](https://mlsanity.com/paper/2503.01411/) (2025)
- [QASA: Quality-Guided K-Adaptive Slot Attention for Unsupervised Object-Centric Learning](https://mlsanity.com/paper/2601.12936/) (2026)
- [Causal Object-Centric Models for Planning with Monte Carlo Tree Search](https://mlsanity.com/paper/2606.14418/) (2026)

## Abstract (arXiv)

We study the problem of binding actions to objects in object-factored world models using action-attention mechanisms. We propose two attention mechanisms for binding actions to objects, soft attention and hard attention, which we evaluate in the context of structured world models for five environments. Our experiments show that hard attention helps contrastively-trained structured world models to learn to separate individual objects in an object-based grid-world environment. Further, we show that soft attention increases performance of factored world models trained on a robotic manipulation task. The learned action attention weights can be used to interpret the factored world model as the attention focuses on the manipulated object in the environment.