# Unlocking the Potential of Similarity Matching: Scalability, Supervision and Pre-training

arXiv: https://arxiv.org/abs/2308.02427 · HTML: https://mlsanity.com/paper/2308.02427/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 17 of its 36 references that are in the map.
- First posted on arXiv 2 Aug 2023.

## Builds on (most cited first, 17 of 17)

- [Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distribution](https://mlsanity.com/paper/2202.10054/) (2022)
- [Equilibrium Propagation: Bridging the Gap Between Energy-Based Models and Backpropagation](https://mlsanity.com/paper/1602.05179/) (2016)
- [Few-Shot Learning via Learning the Representation, Provably](https://mlsanity.com/paper/2002.09434/) (2020)
- [On the Theory of Transfer Learning: The Importance of Task Diversity](https://mlsanity.com/paper/2006.11650/) (2020)
- [Training Neural Networks with Local Error Signals](https://mlsanity.com/paper/1901.06656/) (2019)
- [Greedy Layerwise Learning Can Scale to ImageNet](https://mlsanity.com/paper/1812.11446/) (2018)
- [Dendritic cortical microcircuits approximate the backpropagation algorithm](https://mlsanity.com/paper/1810.11393/) (2018)
- [The HSIC Bottleneck: Deep Learning without Back-Propagation](https://mlsanity.com/paper/1908.01580/) (2019)
- [Kernelized information bottleneck leads to biologically plausible 3-factor Hebbian learning in deep networks](https://mlsanity.com/paper/2006.07123/) (2020)
- [Why do similarity matching objectives lead to Hebbian/anti-Hebbian networks?](https://mlsanity.com/paper/1703.07914/) (2017)
- [Neuroscience-inspired online unsupervised learning algorithms](https://mlsanity.com/paper/1908.01867/) (2019)
- [Constrained Predictive Coding as a Biologically Plausible Model of the Cortical Hierarchy](https://mlsanity.com/paper/2210.15752/) (2022)
- [Structured and Deep Similarity Matching via Structured and Deep Hebbian Networks](https://mlsanity.com/paper/1910.04958/) (2019)
- [A Hebbian/Anti-Hebbian Network for Online Sparse Dictionary Learning Derived from Symmetric Matrix Factorization](https://mlsanity.com/paper/1503.00690/) (2015)
- [Training Deep Architectures Without End-to-End Backpropagation: A Survey on the Provably Optimal Methods](https://mlsanity.com/paper/2101.03419/) (2021)
- [Unsupervised Feature Learning by Deep Sparse Coding](https://mlsanity.com/paper/1312.5783/) (2013)
- [A Neural Network for Semi-Supervised Learning on Manifolds](https://mlsanity.com/paper/1908.08145/) (2019)

## Abstract (arXiv)

While effective, the backpropagation (BP) algorithm exhibits limitations in terms of biological plausibility, computational cost, and suitability for online learning. As a result, there has been a growing interest in developing alternative biologically plausible learning approaches that rely on local learning rules. This study focuses on the primarily unsupervised similarity matching (SM) framework, which aligns with observed mechanisms in biological systems and offers online, localized, and biologically plausible algorithms. i) To scale SM to large datasets, we propose an implementation of Convolutional Nonnegative SM using PyTorch. ii) We introduce a localized supervised SM objective reminiscent of canonical correlation analysis, facilitating stacking SM layers. iii) We leverage the PyTorch implementation for pre-training architectures such as LeNet and compare the evaluation of features against BP-trained models. This work combines biologically plausible algorithms with computational efficiency opening multiple avenues for further explorations.