# Variable Binding for Sparse Distributed Representations: Theory and Applications

arXiv: https://arxiv.org/abs/2009.06734 · HTML: https://mlsanity.com/paper/2009.06734/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 Sep 2020.

## Built on by (most cited first, 27 of 27)

- [A Survey on Hyperdimensional Computing aka Vector Symbolic Architectures, Part I: Models and Data Transformations](https://mlsanity.com/paper/2111.06077/) (2021)
- [A Survey on Hyperdimensional Computing aka Vector Symbolic Architectures, Part II: Applications, Cognitive Models, and Challenges](https://mlsanity.com/paper/2112.15424/) (2021)
- [Computing on Functions Using Randomized Vector Representations](https://mlsanity.com/paper/2109.03429/) (2021)
- [Vector Symbolic Architectures as a Computing Framework for Emerging Hardware](https://mlsanity.com/paper/2106.05268/) (2021)
- [Torchhd: An Open Source Python Library to Support Research on Hyperdimensional Computing and Vector Symbolic Architectures](https://mlsanity.com/paper/2205.09208/) (2022)
- [Neuromorphic Visual Scene Understanding with Resonator Networks](https://mlsanity.com/paper/2208.12880/) (2022)
- [Generalized Learning Vector Quantization for Classification in Randomized Neural Networks and Hyperdimensional Computing](https://mlsanity.com/paper/2106.09821/) (2021)
- [Probabilistic Abduction for Visual Abstract Reasoning via Learning Rules in Vector-symbolic Architectures](https://mlsanity.com/paper/2401.16024/) (2024)
- [Hyperseed: Unsupervised Learning with Vector Symbolic Architectures](https://mlsanity.com/paper/2110.08343/) (2021)
- [Hyperdimensional Computing for Efficient Distributed Classification with Randomized Neural Networks](https://mlsanity.com/paper/2106.00881/) (2021)
- [Deep Phasor Networks: Connecting Conventional and Spiking Neural Networks](https://mlsanity.com/paper/2106.11908/) (2021)
- [Factorizers for Distributed Sparse Block Codes](https://mlsanity.com/paper/2303.13957/) (2023)
- [Streaming Encoding Algorithms for Scalable Hyperdimensional Computing](https://mlsanity.com/paper/2209.09868/) (2022)
- [Capacity Analysis of Vector Symbolic Architectures](https://mlsanity.com/paper/2301.10352/) (2023)
- [Distributed Representations Enable Robust Multi-Timescale Symbolic Computation in Neuromorphic Hardware](https://mlsanity.com/paper/2405.01305/) (2024)
- [On Effects of Compression with Hyperdimensional Computing in Distributed Randomized Neural Networks](https://mlsanity.com/paper/2106.09831/) (2021)
- [Recursive Binding for Similarity-Preserving Hypervector Representations of Sequences](https://mlsanity.com/paper/2201.11691/) (2022)
- [Optimal quadratic binding for relational reasoning in vector symbolic neural architectures](https://mlsanity.com/paper/2204.07186/) (2022)
- [The Hyperdimensional Transform for Distributional Modelling, Regression and Classification](https://mlsanity.com/paper/2311.08150/) (2023)
- [Reframing the Mind-Body Picture: Applying Formal Systems to the Relationship of Mind and Matter](https://mlsanity.com/paper/2404.07719/) (2024)
- [On Design Choices in Similarity-Preserving Sparse Randomized Embeddings](https://mlsanity.com/paper/2501.14741/) (2024)
- [A Short Survey of Systematic Generalization](https://mlsanity.com/paper/2211.11956/) (2022)
- [Developing a Foundation of Vector Symbolic Architectures Using Category Theory](https://mlsanity.com/paper/2501.05368/) (2025)
- [A Grid Cell-Inspired Structured Vector Algebra for Cognitive Maps](https://mlsanity.com/paper/2503.08608/) (2025)
- [High-resolution spatial memory requires grid-cell-like neural codes](https://mlsanity.com/paper/2507.00598/) (2025)
- [Overcoming the Impedance Mismatch: A Theoretical Roadmap for Fusing Foundation Models and Knowledge Graphs](https://mlsanity.com/paper/2606.15656/) (2026)
- [Real-Valued Hyperdimensional Sequence Representations with Hadamard Product Binding and Shift Equivariance](https://mlsanity.com/paper/2608.28334/) (2026)

## Abstract (arXiv)

Symbolic reasoning and neural networks are often considered incompatible approaches. Connectionist models known as Vector Symbolic Architectures (VSAs) can potentially bridge this gap. However, classical VSAs and neural networks are still considered incompatible. VSAs encode symbols by dense pseudo-random vectors, where information is distributed throughout the entire neuron population. Neural networks encode features locally, often forming sparse vectors of neural activation. Following Rachkovskij (2001); Laiho et al. (2015), we explore symbolic reasoning with sparse distributed representations. The core operations in VSAs are dyadic operations between vectors to express variable binding and the representation of sets. Thus, algebraic manipulations enable VSAs to represent and process data structures in a vector space of fixed dimensionality. Using techniques from compressed sensing, we first show that variable binding between dense vectors in VSAs is mathematically equivalent to tensor product binding between sparse vectors, an operation which increases dimensionality. This result implies that dimensionality-preserving binding for general sparse vectors must include a reduction of the tensor matrix into a single sparse vector. Two options for sparsity-preserving variable binding are investigated. One binding method for general sparse vectors extends earlier proposals to reduce the tensor product into a vector, such as circular convolution. The other method is only defined for sparse block-codes, block-wise circular convolution. Our experiments reveal that variable binding for block-codes has ideal properties, whereas binding for general sparse vectors also works, but is lossy, similar to previous proposals. We demonstrate a VSA with sparse block-codes in example applications, cognitive reasoning and classification, and discuss its relevance for neuroscience and neural networks.