# Kernel similarity matching with Hebbian neural networks

arXiv: https://arxiv.org/abs/2204.07475 · HTML: https://mlsanity.com/paper/2204.07475/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 9 of its 32 references that are in the map.
- First posted on arXiv 15 Apr 2022.

## Builds on (most cited first, 9 of 9)

- [On Gradient Descent Ascent for Nonconvex-Concave Minimax Problems](https://mlsanity.com/paper/1906.00331/) (2019)
- [A Hebbian/Anti-Hebbian Neural Network for Linear Subspace Learning: A Derivation from Multidimensional Scaling of Streaming Data](https://mlsanity.com/paper/1503.00669/) (2015)
- [Why do similarity matching objectives lead to Hebbian/anti-Hebbian networks?](https://mlsanity.com/paper/1703.07914/) (2017)
- [A Hebbian/Anti-Hebbian Network Derived from Online Non-Negative Matrix Factorization Can Cluster and Discover Sparse Features](https://mlsanity.com/paper/1503.00680/) (2015)
- [Nonlinear Hebbian learning as a unifying principle in receptive field formation](https://mlsanity.com/paper/1601.00701/) (2016)
- [Structured and Deep Similarity Matching via Structured and Deep Hebbian Networks](https://mlsanity.com/paper/1910.04958/) (2019)
- [A Hebbian/Anti-Hebbian Network for Online Sparse Dictionary Learning Derived from Symmetric Matrix Factorization](https://mlsanity.com/paper/1503.00690/) (2015)
- [A correlation game for unsupervised learning yields computational interpretations of Hebbian excitation, anti-Hebbian inhibition, and synapse elimination](https://mlsanity.com/paper/1704.00646/) (2017)
- [Convergence of gradient descent-ascent analyzed as a Newtonian dynamical system with dissipation](https://mlsanity.com/paper/1903.02536/) (2019)

## Abstract (arXiv)

Recent works have derived neural networks with online correlation-based learning rules to perform textitkernel similarity matching. These works applied existing linear similarity matching algorithms to nonlinear features generated with random Fourier methods. In this paper attempt to perform kernel similarity matching by directly learning the nonlinear features. Our algorithm proceeds by deriving and then minimizing an upper bound for the sum of squared errors between output and input kernel similarities. The construction of our upper bound leads to online correlation-based learning rules which can be implemented with a 1 layer recurrent neural network. In addition to generating high-dimensional linearly separable representations, we show that our upper bound naturally yields representations which are sparse and selective for specific input patterns. We compare the approximation quality of our method to neural random Fourier method and variants of the popular but non-biological "Nystr"om" method for approximating the kernel matrix. Our method appears to be comparable or better than randomly sampled Nystr"om methods when the outputs are relatively low dimensional (although still potentially higher dimensional than the inputs) but less faithful when the outputs are very high dimensional.