# Duality Principle and Biologically Plausible Learning: Connecting the Representer Theorem and Hebbian Learning

arXiv: https://arxiv.org/abs/2309.16687 · HTML: https://mlsanity.com/paper/2309.16687/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 74 references that are in the map.
- First posted on arXiv 2 Aug 2023.

## Builds on (most cited first, 28 of 28)

- [Neural Tangent Kernel: Convergence and Generalization in Neural Networks](https://mlsanity.com/paper/1806.07572/) (2018)
- [Deep Equilibrium Models](https://mlsanity.com/paper/1909.01377/) (2019)
- [On Exact Computation with an Infinitely Wide Neural Net](https://mlsanity.com/paper/1904.11955/) (2019)
- [Spectrum Dependent Learning Curves in Kernel Regression and Wide Neural Networks](https://mlsanity.com/paper/2002.02561/) (2020)
- [Spectral Bias and Task-Model Alignment Explain Generalization in Kernel Regression and Infinitely Wide Neural Networks](https://mlsanity.com/paper/2006.13198/) (2020)
- [Dendritic cortical microcircuits approximate the backpropagation algorithm](https://mlsanity.com/paper/1810.11393/) (2018)
- [The HSIC Bottleneck: Deep Learning without Back-Propagation](https://mlsanity.com/paper/1908.01580/) (2019)
- [Towards deep learning with segregated dendrites](https://mlsanity.com/paper/1610.00161/) (2016)
- [Towards Biologically Plausible Deep Learning](https://mlsanity.com/paper/1502.04156/) (2015)
- [Biologically Motivated Algorithms for Propagating Local Target Representations](https://mlsanity.com/paper/1805.11703/) (2018)
- [Credit Assignment in Neural Networks through Deep Feedback Control](https://mlsanity.com/paper/2106.07887/) (2021)
- [A Hebbian/Anti-Hebbian Neural Network for Linear Subspace Learning: A Derivation from Multidimensional Scaling of Streaming Data](https://mlsanity.com/paper/1503.00669/) (2015)
- [Kernelized information bottleneck leads to biologically plausible 3-factor Hebbian learning in deep networks](https://mlsanity.com/paper/2006.07123/) (2020)
- [Beyond Backprop: Online Alternating Minimization with Auxiliary Variables](https://mlsanity.com/paper/1806.09077/) (2018)
- [Why do similarity matching objectives lead to Hebbian/anti-Hebbian networks?](https://mlsanity.com/paper/1703.07914/) (2017)
- [Neuroscience-inspired online unsupervised learning algorithms](https://mlsanity.com/paper/1908.01867/) (2019)
- [Constrained Predictive Coding as a Biologically Plausible Model of the Cortical Hierarchy](https://mlsanity.com/paper/2210.15752/) (2022)
- [The least-control principle for local learning at equilibrium](https://mlsanity.com/paper/2207.01332/) (2022)
- [A Normative Theory of Adaptive Dimensionality Reduction in Neural Networks](https://mlsanity.com/paper/1511.09426/) (2015)
- [A Hebbian/Anti-Hebbian Network Derived from Online Non-Negative Matrix Factorization Can Cluster and Discover Sparse Features](https://mlsanity.com/paper/1503.00680/) (2015)
- [Structured and Deep Similarity Matching via Structured and Deep Hebbian Networks](https://mlsanity.com/paper/1910.04958/) (2019)
- [A biologically plausible neural network for multi-channel Canonical Correlation Analysis](https://mlsanity.com/paper/2010.00525/) (2020)
- [Biologically plausible single-layer networks for nonnegative independent component analysis](https://mlsanity.com/paper/2010.12632/) (2020)
- [A biologically plausible neural network for Slow Feature Analysis](https://mlsanity.com/paper/2010.12644/) (2020)
- [A Normative and Biologically Plausible Algorithm for Independent Component Analysis](https://mlsanity.com/paper/2111.08858/) (2021)
- [A simple normative network approximates local non-Hebbian learning in the cortex](https://mlsanity.com/paper/2010.12660/) (2020)
- [Normative framework for deriving neural networks with multi-compartmental neurons and non-Hebbian plasticity](https://mlsanity.com/paper/2302.10051/) (2023)
- [A Similarity-preserving Neural Network Trained on Transformed Images Recapitulates Salient Features of the Fly Motion Detection Circuit](https://mlsanity.com/paper/2102.05503/) (2021)

## Built on by (most cited first, 2 of 2)

- [Brain-Inspired Machine Intelligence: A Survey of Neurobiologically-Plausible Credit Assignment](https://mlsanity.com/paper/2312.09257/) (2023)
- [Curl Descent: Non-Gradient Learning Dynamics with Sign-Diverse Plasticity](https://mlsanity.com/paper/2510.02765/) (2025)

## Abstract (arXiv)

A normative approach called Similarity Matching was recently introduced for deriving and understanding the algorithmic basis of neural computation focused on unsupervised problems. It involves deriving algorithms from computational objectives and evaluating their compatibility with anatomical and physiological observations. In particular, it introduces neural architectures by considering dual alternatives instead of primal formulations of popular models such as PCA. However, its connection to the Representer theorem remains unexplored. In this work, we propose to use teachings from this approach to explore supervised learning algorithms and clarify the notion of Hebbian learning. We examine regularized supervised learning and elucidate the emergence of neural architecture and additive versus multiplicative update rules. In this work, we focus not on developing new algorithms but on showing that the Representer theorem offers the perfect lens to study biologically plausible learning algorithms. We argue that many past and current advancements in the field rely on some form of dual formulation to introduce biological plausibility. In short, as long as a dual formulation exists, it is possible to derive biologically plausible algorithms. Our work sheds light on the pivotal role of the Representer theorem in advancing our comprehension of neural computation.