# Dance of SNN and ANN: Solving binding problem by combining spike timing and reconstructive attention

arXiv: https://arxiv.org/abs/2211.06027 · HTML: https://mlsanity.com/paper/2211.06027/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 78 references that are in the map.
- First posted on arXiv 11 Nov 2022.

## Builds on (most cited first, 26 of 26)

- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Supervised Contrastive Learning](https://mlsanity.com/paper/2004.11362/) (2020)
- [Rich feature hierarchies for accurate object detection and semantic segmentation](https://mlsanity.com/paper/1311.2524/) (2013)
- [Object-Centric Learning with Slot Attention](https://mlsanity.com/paper/2006.15055/) (2020)
- [CLEVRER: CoLlision Events for Video REpresentation and Reasoning](https://mlsanity.com/paper/1910.01442/) (2019)
- [MONet: Unsupervised Scene Decomposition and Representation](https://mlsanity.com/paper/1901.11390/) (2019)
- [Multi-Object Representation Learning with Iterative Variational Inference](https://mlsanity.com/paper/1903.00450/) (2019)
- [Towards a Definition of Disentangled Representations](https://mlsanity.com/paper/1812.02230/) (2018)
- [On the Binding Problem in Artificial Neural Networks](https://mlsanity.com/paper/2012.05208/) (2020)
- [GENESIS: Generative Scene Inference and Sampling with Object-Centric Latent Representations](https://mlsanity.com/paper/1907.13052/) (2019)
- [Attend, Infer, Repeat: Fast Scene Understanding with Generative Models](https://mlsanity.com/paper/1603.08575/) (2016)
- [SPACE: Unsupervised Object-Oriented Scene Representation via Spatial Attention and Decomposition](https://mlsanity.com/paper/2001.02407/) (2020)
- [Sequential Attend, Infer, Repeat: Generative Modelling of Moving Objects](https://mlsanity.com/paper/1806.01794/) (2018)
- [Neural Expectation Maximization](https://mlsanity.com/paper/1708.03498/) (2017)
- [SCALOR: Generative World Models with Scalable Object Representations](https://mlsanity.com/paper/1910.02384/) (2019)
- [Relational Neural Expectation Maximization: Unsupervised Discovery of Objects and their Interactions](https://mlsanity.com/paper/1802.10353/) (2018)
- [Visual Interaction Networks](https://mlsanity.com/paper/1706.01433/) (2017)
- [Unsupervised Learning of Object Structure and Dynamics from Videos](https://mlsanity.com/paper/1906.07889/) (2019)
- [Tagger: Deep Unsupervised Perceptual Grouping](https://mlsanity.com/paper/1606.06724/) (2016)
- [Deep Object-Centric Representations for Generalizable Robot Learning](https://mlsanity.com/paper/1708.04225/) (2017)
- [Complex-Valued Autoencoders for Object Discovery](https://mlsanity.com/paper/2204.02075/) (2022)
- [Graph Density-Aware Losses for Novel Compositions in Scene Graph Generation](https://mlsanity.com/paper/2005.08230/) (2020)
- [Neuronal Synchrony in Complex-Valued Deep Networks](https://mlsanity.com/paper/1312.6115/) (2013)
- [Stochastic Prediction of Multi-Agent Interactions from Partial Observations](https://mlsanity.com/paper/1902.09641/) (2019)
- [Binding via Reconstruction Clustering](https://mlsanity.com/paper/1511.06418/) (2015)
- [Unsupervised Object Keypoint Learning using Local Spatial Predictability](https://mlsanity.com/paper/2011.12930/) (2020)

## Built on by (most cited first, 3 of 3)

- [Binding Dynamics in Rotating Features](https://mlsanity.com/paper/2402.05627/) (2024)
- [Temporal Spiking Neural Networks with Synaptic Delay for Graph Reasoning](https://mlsanity.com/paper/2405.16851/) (2024)
- [Kuramoto Oscillatory Phase Encoding: Neuro-inspired Synchronization for Improved Learning Efficiency](https://mlsanity.com/paper/2604.07904/) (2026)

## Abstract (arXiv)

The binding problem is one of the fundamental challenges that prevent the artificial neural network (ANNs) from a compositional understanding of the world like human perception, because disentangled and distributed representations of generative factors can interfere and lead to ambiguity when complex data with multiple objects are presented. In this paper, we propose a brain-inspired hybrid neural network (HNN) that introduces temporal binding theory originated from neuroscience into ANNs by integrating spike timing dynamics (via spiking neural networks, SNNs) with reconstructive attention (by ANNs). Spike timing provides an additional dimension for grouping, while reconstructive feedback coordinates the spikes into temporal coherent states. Through iterative interaction of ANN and SNN, the model continuously binds multiple objects at alternative synchronous firing times in the SNN coding space. The effectiveness of the model is evaluated on synthetic datasets of binary images. By visualization and analysis, we demonstrate that the binding is explainable, soft, flexible, and hierarchical. Notably, the model is trained on single object datasets without explicit supervision on grouping, but successfully binds multiple objects on test datasets, showing its compositional generalization capability. Further results show its binding ability in dynamic situations.