# Neuromorphic Visual Scene Understanding with Resonator Networks

arXiv: https://arxiv.org/abs/2208.12880 · HTML: https://mlsanity.com/paper/2208.12880/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 97 references that are in the map.
- First posted on arXiv 26 Aug 2022.

## Builds on (most cited first, 30 of 32)

- [Towards Deep Learning Models Resistant to Adversarial Attacks](https://mlsanity.com/paper/1706.06083/) (2017)
- [Intriguing properties of neural networks](https://mlsanity.com/paper/1312.6199/) (2013)
- [You Only Look Once: Unified, Real-Time Object Detection](https://mlsanity.com/paper/1506.02640/) (2015)
- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [Shortcut Learning in Deep Neural Networks](https://mlsanity.com/paper/2004.07780/) (2020)
- [Adversarial examples in the physical world](https://mlsanity.com/paper/1607.02533/) (2016)
- [Event-based Vision: A Survey](https://mlsanity.com/paper/1904.08405/) (2019)
- [Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representations](https://mlsanity.com/paper/1811.12359/) (2018)
- [Deep Neural Networks are Easily Fooled: High Confidence Predictions for Unrecognizable Images](https://mlsanity.com/paper/1412.1897/) (2014)
- [Disentangling by Factorising](https://mlsanity.com/paper/1802.05983/) (2018)
- [Variational Autoencoders and Nonlinear ICA: A Unifying Framework](https://mlsanity.com/paper/1907.04809/) (2019)
- [Why do deep convolutional networks generalize so poorly to small image transformations?](https://mlsanity.com/paper/1805.12177/) (2018)
- [Strike (with) a Pose: Neural Networks Are Easily Fooled by Strange Poses of Familiar Objects](https://mlsanity.com/paper/1811.11553/) (2018)
- [A Survey on Hyperdimensional Computing aka Vector Symbolic Architectures, Part I: Models and Data Transformations](https://mlsanity.com/paper/2111.06077/) (2021)
- [A Survey on Hyperdimensional Computing aka Vector Symbolic Architectures, Part II: Applications, Cognitive Models, and Challenges](https://mlsanity.com/paper/2112.15424/) (2021)
- [Excessive Invariance Causes Adversarial Vulnerability](https://mlsanity.com/paper/1811.00401/) (2018)
- [Visual Representation Learning Does Not Generalize Strongly Within the Same Domain](https://mlsanity.com/paper/2107.08221/) (2021)
- [Assaying Out-Of-Distribution Generalization in Transfer Learning](https://mlsanity.com/paper/2207.09239/) (2022)
- [A scalable multi-core architecture with heterogeneous memory structures for Dynamic Neuromorphic Asynchronous Processors (DYNAPs)](https://mlsanity.com/paper/1708.04198/) (2017)
- [Memory and information processing in neuromorphic systems](https://mlsanity.com/paper/1506.03264/) (2015)
- [Temporal Coding in Spiking Neural Networks with Alpha Synaptic Function: Learning with Backpropagation](https://mlsanity.com/paper/1907.13223/) (2019)
- [Neural Manifold Clustering and Embedding](https://mlsanity.com/paper/2201.10000/) (2022)
- [Computing on Functions Using Randomized Vector Representations](https://mlsanity.com/paper/2109.03429/) (2021)
- [A theory of sequence indexing and working memory in recurrent neural networks](https://mlsanity.com/paper/1803.00412/) (2018)
- [Variable Binding for Sparse Distributed Representations: Theory and Applications](https://mlsanity.com/paper/2009.06734/) (2020)
- [Robust computation with rhythmic spike patterns](https://mlsanity.com/paper/1901.07718/) (2019)
- [Disentangling images with Lie group transformations and sparse coding](https://mlsanity.com/paper/2012.12071/) (2020)
- [Efficient inference in occlusion-aware generative models of images](https://mlsanity.com/paper/1511.06362/) (2015)
- [DiagViB-6: A Diagnostic Benchmark Suite for Vision Models in the Presence of Shortcut and Generalization Opportunities](https://mlsanity.com/paper/2108.05779/) (2021)

## Built on by (most cited first, 17 of 17)

- [Applications of Spiking Neural Networks in Visual Place Recognition](https://mlsanity.com/paper/2311.13186/) (2023)
- [Computing with Residue Numbers in High-Dimensional Representation](https://mlsanity.com/paper/2311.04872/) (2023)
- [Visual Odometry with Neuromorphic Resonator Networks](https://mlsanity.com/paper/2209.02000/) (2022)
- [Learning and generalization of compositional representations of visual scenes](https://mlsanity.com/paper/2303.13691/) (2023)
- [Factorizers for Distributed Sparse Block Codes](https://mlsanity.com/paper/2303.13957/) (2023)
- [Vector-Symbolic Architecture for Event-Based Optical Flow](https://mlsanity.com/paper/2405.08300/) (2024)
- [Compositional Factorization of Visual Scenes with Convolutional Sparse Coding and Resonator Networks](https://mlsanity.com/paper/2404.19126/) (2024)
- [Distributed Representations Enable Robust Multi-Timescale Symbolic Computation in Neuromorphic Hardware](https://mlsanity.com/paper/2405.01305/) (2024)
- [On the Role of Noise in Factorizers for Disentangling Distributed Representations](https://mlsanity.com/paper/2412.00354/) (2024)
- [A Grid Cell-Inspired Structured Vector Algebra for Cognitive Maps](https://mlsanity.com/paper/2503.08608/) (2025)
- [Heterogeneous computing platform for real-time robotics](https://mlsanity.com/paper/2601.09755/) (2026)
- [Coupled Inference in Diffusion Models for Semantic Decomposition](https://mlsanity.com/paper/2602.09983/) (2026)
- [Generalized multi-object classification and tracking with sparse feature resonator networks](https://mlsanity.com/paper/2603.22539/) (2026)
- [Robot Arm Control via Cognitive Map Learners](https://mlsanity.com/paper/2603.26773/) (2026)
- [HyperSpace: A Generalized Framework for Spatial Encoding in Hyperdimensional Representations](https://mlsanity.com/paper/2604.15113/) (2026)
- [SRMU: Relevance-Gated Updates for Streaming Hyperdimensional Memories](https://mlsanity.com/paper/2604.15121/) (2026)
- [Disentanglement with Holographic Reduced Representations](https://mlsanity.com/paper/2606.09725/) (2026)

## Abstract (arXiv)

Analyzing a visual scene by inferring the configuration of a generative model is widely considered the most flexible and generalizable approach to scene understanding. Yet, one major problem is the computational challenge of the inference procedure, involving a combinatorial search across object identities and poses. Here we propose a neuromorphic solution exploiting three key concepts: (1) a computational framework based on Vector Symbolic Architectures (VSA) with complex-valued vectors; (2) the design of Hierarchical Resonator Networks (HRN) to factorize the non-commutative transforms translation and rotation in visual scenes; (3) the design of a multi-compartment spiking phasor neuron model for implementing complex-valued resonator networks on neuromorphic hardware. The VSA framework uses vector binding operations to form a generative image model in which binding acts as the equivariant operation for geometric transformations. A scene can, therefore, be described as a sum of vector products, which can then be efficiently factorized by a resonator network to infer objects and their poses. The HRN features a partitioned architecture in which vector binding is equivariant for horizontal and vertical translation within one partition and for rotation and scaling within the other partition. The spiking neuron model allows mapping the resonator network onto efficient and low-power neuromorphic hardware. Our approach is demonstrated on synthetic scenes composed of simple 2D shapes undergoing rigid geometric transformations and color changes. A companion paper demonstrates the same approach in real-world application scenarios for machine vision and robotics.