# Kernelized information bottleneck leads to biologically plausible 3-factor Hebbian learning in deep networks

arXiv: https://arxiv.org/abs/2006.07123 · HTML: https://mlsanity.com/paper/2006.07123/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 Jun 2020.

## Built on by (most cited first, 25 of 25)

- [Self-Supervised Learning with Kernel Dependence Maximization](https://mlsanity.com/paper/2106.08320/) (2021)
- [Efficient Knowledge Distillation from Model Checkpoints](https://mlsanity.com/paper/2210.06458/) (2022)
- [SoftHebb: Bayesian Inference in Unsupervised Hebbian Soft Winner-Take-All Networks](https://mlsanity.com/paper/2107.05747/) (2021)
- [Neuro-Modulated Hebbian Learning for Fully Test-Time Adaptation](https://mlsanity.com/paper/2303.00914/) (2023)
- [Towards Biologically Plausible Convolutional Networks](https://mlsanity.com/paper/2106.13031/) (2021)
- [Spiking Neural Networks and Bio-Inspired Supervised Deep Learning: A Survey](https://mlsanity.com/paper/2307.16235/) (2023)
- [End-to-End Training Induces Information Bottleneck through Layer-Role Differentiation: A Comparative Analysis with Layer-wise Training](https://mlsanity.com/paper/2402.09050/) (2024)
- [Short-Term Plasticity Neurons Learning to Learn and Forget](https://mlsanity.com/paper/2206.14048/) (2022)
- [Training Deep Architectures Without End-to-End Backpropagation: A Survey on the Provably Optimal Methods](https://mlsanity.com/paper/2101.03419/) (2021)
- [Flexible Phase Dynamics for Bio-Plausible Contrastive Learning](https://mlsanity.com/paper/2302.12431/) (2023)
- [Synaptic Weight Distributions Depend on the Geometry of Plasticity](https://mlsanity.com/paper/2305.19394/) (2023)
- [Diversity-enhancing Generative Network for Few-shot Hypothesis Adaptation](https://mlsanity.com/paper/2307.05948/) (2023)
- [Explaining Grokking and Information Bottleneck through Neural Collapse Emergence](https://mlsanity.com/paper/2509.20829/) (2025)
- [Deep Layer-wise Networks Have Closed-Form Weights](https://mlsanity.com/paper/2202.01210/) (2022)
- [Improving Performance in Continual Learning Tasks using Bio-Inspired Architectures](https://mlsanity.com/paper/2308.04539/) (2023)
- [Duality Principle and Biologically Plausible Learning: Connecting the Representer Theorem and Hebbian Learning](https://mlsanity.com/paper/2309.16687/) (2023)
- [Representation Learning in a Decomposed Encoder Design for Bio-inspired Hebbian Learning](https://mlsanity.com/paper/2401.08603/) (2023)
- [Information Bottleneck-Based Hebbian Learning Rule Naturally Ties Working Memory and Synaptic Updates](https://mlsanity.com/paper/2111.13187/) (2021)
- [Biology-inspired joint distribution neurons based on Hierarchical Correlation Reconstruction allowing for multidirectional propagation of values and densities](https://mlsanity.com/paper/2405.05097/) (2024)
- [Conditioned Direct Feedback Alignment via Activity and Error Geometry](https://mlsanity.com/paper/2607.18574/) (2026)
- [Unlocking the Potential of Similarity Matching: Scalability, Supervision and Pre-training](https://mlsanity.com/paper/2308.02427/) (2023)
- [Extreme Image Transformations Facilitate Robust Latent Object Representations](https://mlsanity.com/paper/2310.07725/) (2023)
- [An experimental comparative study of backpropagation and alternatives for training binary neural networks for image classification](https://mlsanity.com/paper/2408.04460/) (2024)
- [Linear cost mutual information estimation and independence test of similar performance as HSIC](https://mlsanity.com/paper/2508.18338/) (2025)
- [Information-Theoretic Greedy Layer-wise Training for Traffic Sign Recognition](https://mlsanity.com/paper/2510.27651/) (2025)

## Abstract (arXiv)

The state-of-the art machine learning approach to training deep neural networks, backpropagation, is implausible for real neural networks: neurons need to know their outgoing weights; training alternates between a bottom-up forward pass (computation) and a top-down backward pass (learning); and the algorithm often needs precise labels of many data points. Biologically plausible approximations to backpropagation, such as feedback alignment, solve the weight transport problem, but not the other two. Thus, fully biologically plausible learning rules have so far remained elusive. Here we present a family of learning rules that does not suffer from any of these problems. It is motivated by the information bottleneck principle (extended with kernel methods), in which networks learn to compress the input as much as possible without sacrificing prediction of the output. The resulting rules have a 3-factor Hebbian structure: they require pre- and post-synaptic firing rates and an error signal - the third factor - consisting of a global teaching signal and a layer-specific term, both available without a top-down pass. They do not require precise labels; instead, they rely on the similarity between pairs of desired outputs. Moreover, to obtain good performance on hard problems and retain biological plausibility, our rules need divisive normalization - a known feature of biological networks. Finally, simulations show that our rules perform nearly as well as backpropagation on image classification tasks.