# Learning and generalization of compositional representations of visual scenes

arXiv: https://arxiv.org/abs/2303.13691 · HTML: https://mlsanity.com/paper/2303.13691/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 3 of its 13 references that are in the map.
- First posted on arXiv 23 Mar 2023.

## Builds on (most cited first, 3 of 3)

- [A theory of sequence indexing and working memory in recurrent neural networks](https://mlsanity.com/paper/1803.00412/) (2018)
- [Neuromorphic Visual Scene Understanding with Resonator Networks](https://mlsanity.com/paper/2208.12880/) (2022)
- [Disentangling images with Lie group transformations and sparse coding](https://mlsanity.com/paper/2012.12071/) (2020)

## Built on by (most cited first, 5 of 5)

- [Compositional Generalization from First Principles](https://mlsanity.com/paper/2307.05596/) (2023)
- [Provable Compositional Generalization for Object-Centric Learning](https://mlsanity.com/paper/2310.05327/) (2023)
- [Deciphering the Role of Representation Disentanglement: Investigating Compositional Generalization in CLIP Models](https://mlsanity.com/paper/2407.05897/) (2024)
- [Successes and Limitations of Object-centric Models at Compositional Generalisation](https://mlsanity.com/paper/2412.18743/) (2024)
- [Generalized multi-object classification and tracking with sparse feature resonator networks](https://mlsanity.com/paper/2603.22539/) (2026)

## Abstract (arXiv)

Complex visual scenes that are composed of multiple objects, each with attributes, such as object name, location, pose, color, etc., are challenging to describe in order to train neural networks. Usually,deep learning networks are trained supervised by categorical scene descriptions. The common categorical description of a scene contains the names of individual objects but lacks information about other attributes. Here, we use distributed representations of object attributes and vector operations in a vector symbolic architecture to create a full compositional description of a scene in a high-dimensional vector. To control the scene composition, we use artificial images composed of multiple, translated and colored MNIST digits. In contrast to learning category labels, here we train deep neural networks to output the full compositional vector description of an input image. The output of the deep network can then be interpreted by a VSA resonator network, to extract object identity or other properties of indiviual objects. We evaluate the performance and generalization properties of the system on randomly generated scenes. Specifically, we show that the network is able to learn the task and generalize to unseen seen digit shapes and scene configurations. Further, the generalisation ability of the trained model is limited. For example, with a gap in the training data, like an object not shown in a particular image location during training, the learning does not automatically fill this gap.