# Non-linear Neurons with Human-like Apical Dendrite Activations

arXiv: https://arxiv.org/abs/2003.03229 · HTML: https://mlsanity.com/paper/2003.03229/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Feb 2020.

## Built on by (most cited first, 5 of 5)

- [Adaptive Rational Activations to Boost Deep Reinforcement Learning](https://mlsanity.com/paper/2102.09407/) (2021)
- [SepTr: Separable Transformer for Audio Spectrogram Processing](https://mlsanity.com/paper/2203.09581/) (2022)
- [Self-paced ensemble learning for speech and audio classification](https://mlsanity.com/paper/2103.11988/) (2021)
- [Teacher-Student Training and Triplet Loss to Reduce the Effect of Drastic Face Occlusion](https://mlsanity.com/paper/2111.10561/) (2021)
- [Towards NeuroAI: Introducing Neuronal Diversity into Artificial Neural Networks](https://mlsanity.com/paper/2301.09245/) (2023)

## Abstract (arXiv)

In order to classify linearly non-separable data, neurons are typically organized into multi-layer neural networks that are equipped with at least one hidden layer. Inspired by some recent discoveries in neuroscience, we propose a new model of artificial neuron along with a novel activation function enabling the learning of nonlinear decision boundaries using a single neuron. We show that a standard neuron followed by our novel apical dendrite activation (ADA) can learn the XOR logical function with 100% accuracy. Furthermore, we conduct experiments on six benchmark data sets from computer vision, signal processing and natural language processing, i.e. MOROCO, UTKFace, CREMA-D, Fashion-MNIST, Tiny ImageNet and ImageNet, showing that the ADA and the leaky ADA functions provide superior results to Rectified Linear Units (ReLU), leaky ReLU, RBF and Swish, for various neural network architectures, e.g. one-hidden-layer or two-hidden-layer multi-layer perceptrons (MLPs) and convolutional neural networks (CNNs) such as LeNet, VGG, ResNet and Character-level CNN. We obtain further performance improvements when we change the standard model of the neuron with our pyramidal neuron with apical dendrite activations (PyNADA). Our code is available at: https://github.com/raduionescu/pynada.