# A theory of sequence indexing and working memory in recurrent neural networks

arXiv: https://arxiv.org/abs/1803.00412 · HTML: https://mlsanity.com/paper/1803.00412/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 Feb 2018.

## Built on by (most cited first, 29 of 29)

- [A Survey on Hyperdimensional Computing aka Vector Symbolic Architectures, Part I: Models and Data Transformations](https://mlsanity.com/paper/2111.06077/) (2021)
- [A Survey on Hyperdimensional Computing aka Vector Symbolic Architectures, Part II: Applications, Cognitive Models, and Challenges](https://mlsanity.com/paper/2112.15424/) (2021)
- [Computing on Functions Using Randomized Vector Representations](https://mlsanity.com/paper/2109.03429/) (2021)
- [Vector Symbolic Architectures as a Computing Framework for Emerging Hardware](https://mlsanity.com/paper/2106.05268/) (2021)
- [Hyperdimensional computing as a framework for systematic aggregation of image descriptors](https://mlsanity.com/paper/2101.07720/) (2021)
- [Learning with Holographic Reduced Representations](https://mlsanity.com/paper/2109.02157/) (2021)
- [Understanding Hyperdimensional Computing for Parallel Single-Pass Learning](https://mlsanity.com/paper/2202.04805/) (2022)
- [Torchhd: An Open Source Python Library to Support Research on Hyperdimensional Computing and Vector Symbolic Architectures](https://mlsanity.com/paper/2205.09208/) (2022)
- [Neuromorphic Visual Scene Understanding with Resonator Networks](https://mlsanity.com/paper/2208.12880/) (2022)
- [HDC-MiniROCKET: Explicit Time Encoding in Time Series Classification with Hyperdimensional Computing](https://mlsanity.com/paper/2202.08055/) (2022)
- [Generalized Learning Vector Quantization for Classification in Randomized Neural Networks and Hyperdimensional Computing](https://mlsanity.com/paper/2106.09821/) (2021)
- [Hyperseed: Unsupervised Learning with Vector Symbolic Architectures](https://mlsanity.com/paper/2110.08343/) (2021)
- [Hyperdimensional Computing for Efficient Distributed Classification with Randomized Neural Networks](https://mlsanity.com/paper/2106.00881/) (2021)
- [Computing with Residue Numbers in High-Dimensional Representation](https://mlsanity.com/paper/2311.04872/) (2023)
- [Generalized Holographic Reduced Representations](https://mlsanity.com/paper/2405.09689/) (2024)
- [Learning and generalization of compositional representations of visual scenes](https://mlsanity.com/paper/2303.13691/) (2023)
- [Streaming Encoding Algorithms for Scalable Hyperdimensional Computing](https://mlsanity.com/paper/2209.09868/) (2022)
- [High-dimensional distributed semantic spaces for utterances](https://mlsanity.com/paper/2104.00424/) (2021)
- [On Effects of Compression with Hyperdimensional Computing in Distributed Randomized Neural Networks](https://mlsanity.com/paper/2106.09831/) (2021)
- [Recursive Binding for Similarity-Preserving Hypervector Representations of Sequences](https://mlsanity.com/paper/2201.11691/) (2022)
- [Generalized Key-Value Memory to Flexibly Adjust Redundancy in Memory-Augmented Networks](https://mlsanity.com/paper/2203.06223/) (2022)
- [Optimal quadratic binding for relational reasoning in vector symbolic neural architectures](https://mlsanity.com/paper/2204.07186/) (2022)
- [The Hyperdimensional Transform: a Holographic Representation of Functions](https://mlsanity.com/paper/2310.16065/) (2023)
- [Self-Attention Based Semantic Decomposition in Vector Symbolic Architectures](https://mlsanity.com/paper/2403.13218/) (2024)
- [On Design Choices in Similarity-Preserving Sparse Randomized Embeddings](https://mlsanity.com/paper/2501.14741/) (2024)
- [Hyperdimensional Quantum Factorization](https://mlsanity.com/paper/2406.11889/) (2024)
- [Autonomous Learning with High-Dimensional Computing Architecture Similar to von Neumann's](https://mlsanity.com/paper/2503.23608/) (2025)
- [SRMU: Relevance-Gated Updates for Streaming Hyperdimensional Memories](https://mlsanity.com/paper/2604.15121/) (2026)
- [The Limits of Binding in Dual Encoders](https://mlsanity.com/paper/2608.15971/) (2026)

## Abstract (arXiv)

To accommodate structured approaches of neural computation, we propose a class of recurrent neural networks for indexing and storing sequences of symbols or analog data vectors. These networks with randomized input weights and orthogonal recurrent weights implement coding principles previously described in vector symbolic architectures (VSA), and leverage properties of reservoir computing. In general, the storage in reservoir computing is lossy and crosstalk noise limits the retrieval accuracy and information capacity. A novel theory to optimize memory performance in such networks is presented and compared with simulation experiments. The theory describes linear readout of analog data, and readout with winner-take-all error correction of symbolic data as proposed in VSA models. We find that diverse VSA models from the literature have universal performance properties, which are superior to what previous analyses predicted. Further, we propose novel VSA models with the statistically optimal Wiener filter in the readout that exhibit much higher information capacity, in particular for storing analog data. The presented theory also applies to memory buffers, networks with gradual forgetting, which can operate on infinite data streams without memory overflow. Interestingly, we find that different forgetting mechanisms, such as attenuating recurrent weights or neural nonlinearities, produce very similar behavior if the forgetting time constants are aligned. Such models exhibit extensive capacity when their forgetting time constant is optimized for given noise conditions and network size. These results enable the design of new types of VSA models for the online processing of data streams.