# Biology-inspired joint distribution neurons based on Hierarchical Correlation Reconstruction allowing for multidirectional propagation of values and densities

arXiv: https://arxiv.org/abs/2405.05097 · HTML: https://mlsanity.com/paper/2405.05097/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 31 references that are in the map.
- First posted on arXiv 8 May 2024.

## Builds on (most cited first, 16 of 16)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Mamba: Linear-Time Sequence Modeling with Selective State Spaces](https://mlsanity.com/paper/2312.00752/) (2023)
- [DINO: DETR with Improved DeNoising Anchor Boxes for End-to-End Object Detection](https://mlsanity.com/paper/2203.03605/) (2022)
- [Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecture](https://mlsanity.com/paper/2301.08243/) (2023)
- [Deep Learning and the Information Bottleneck Principle](https://mlsanity.com/paper/1503.02406/) (2015)
- [KAN: Kolmogorov-Arnold Networks](https://mlsanity.com/paper/2404.19756/) (2024)
- [Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets](https://mlsanity.com/paper/2201.02177/) (2022)
- [The HSIC Bottleneck: Deep Learning without Back-Propagation](https://mlsanity.com/paper/1908.01580/) (2019)
- [Kernelized information bottleneck leads to biologically plausible 3-factor Hebbian learning in deep networks](https://mlsanity.com/paper/2006.07123/) (2020)
- [Hierarchical correlation reconstruction with missing data, for example for biology-inspired neuron](https://mlsanity.com/paper/1804.06218/) (2018)
- [Modelling bid-ask spread conditional distributions using hierarchical correlation reconstruction](https://mlsanity.com/paper/1911.02361/) (2019)
- [Exploiting statistical dependencies of time series with hierarchical correlation reconstruction](https://mlsanity.com/paper/1807.04119/) (2018)
- [Predicting conditional probability distributions of redshifts of Active Galactic Nuclei using Hierarchical Correlation Reconstruction](https://mlsanity.com/paper/2206.06194/) (2022)
- [Rapid parametric density estimation](https://mlsanity.com/paper/1702.02144/) (2017)
- [Fast optimization of common basis for matrix set through Common Singular Value Decomposition](https://mlsanity.com/paper/2204.08242/) (2022)

## Built on by (most cited first, 1 of 1)

- [Higher order PCA-like rotation-invariant features for detailed shape descriptors modulo rotation](https://mlsanity.com/paper/2601.03326/) (2026)

## Abstract (arXiv)

Recently a million of biological neurons (BNN) has turned out better from modern RL methods in playing Pong~citeRL, reminding they are still qualitatively superior e.g. in learning, flexibility and robustness - suggesting to try to improve current artificial e.g. MLP/KAN for better agreement with biological. There is proposed extension of KAN approach to neurons containing model of local joint distribution: ρ(x)=sumj∈ B aj fj(x) for x ∈ [0,1]d, adding interpretation and information flow control to KAN, and allowing to gradually add missing 3 basic properties of biological: 1) biological axons propagate in both directions~citeaxon, while current artificial are focused on unidirectional propagation - joint distribution neurons can repair by substituting some variables to get conditional values/distributions for the remaining. 2) Animals show risk avoidance~citerisk requiring to process variance, and generally real world rather needs probabilistic models - the proposed can predict and propagate also distributions as vectors of moments: (expected value, variance) or higher. 3) biological neurons require local training, and beside backpropagation, the proposed allows many additional ways, like direct training, through tensor decomposition, or finally local and promising: information bottleneck. Proposed approach is very general, can be also used as extension of softmax in embeddings of e.g. transformer, JEPA, Mamba, suggesting interpretation that features are mixed moments of joint density of real-world properties.