# Logical Activation Functions: Logit-space equivalents of Probabilistic Boolean Operators

arXiv: https://arxiv.org/abs/2110.11940 · HTML: https://mlsanity.com/paper/2110.11940/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 73 references that are in the map.
- First posted on arXiv 22 Oct 2021.

## Builds on (most cited first, 27 of 27)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Gaussian Error Linear Units (GELUs)](https://mlsanity.com/paper/1606.08415/) (2016)
- [Identity Mappings in Deep Residual Networks](https://mlsanity.com/paper/1603.05027/) (2016)
- [CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning](https://mlsanity.com/paper/1612.06890/) (2016)
- [Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)](https://mlsanity.com/paper/1511.07289/) (2015)
- [Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learning](https://mlsanity.com/paper/1702.03118/) (2017)
- [AutoAugment: Learning Augmentation Policies from Data](https://mlsanity.com/paper/1805.09501/) (2018)
- [Self-Normalizing Neural Networks](https://mlsanity.com/paper/1706.02515/) (2017)
- [A disciplined approach to neural network hyper-parameters: Part 1 -- learning rate, batch size, momentum, and weight decay](https://mlsanity.com/paper/1803.09820/) (2018)
- [RAVEN: A Dataset for Relational and Analogical Visual rEasoNing](https://mlsanity.com/paper/1903.02741/) (2019)
- [Sorting out Lipschitz function approximation](https://mlsanity.com/paper/1811.05381/) (2018)
- [Super-Convergence: Very Fast Training of Neural Networks Using Large Learning Rates](https://mlsanity.com/paper/1708.07120/) (2017)
- [Maxout Networks](https://mlsanity.com/paper/1302.4389/) (2013)
- [Measuring abstract reasoning in neural networks](https://mlsanity.com/paper/1807.04225/) (2018)
- [Visualizing and Understanding Recurrent Networks](https://mlsanity.com/paper/1506.02078/) (2015)
- [Learning to Generate Reviews and Discovering Sentiment](https://mlsanity.com/paper/1704.01444/) (2017)
- [Mish: A Self Regularized Non-Monotonic Activation Function](https://mlsanity.com/paper/1908.08681/) (2019)
- [Task-Driven Modular Networks for Zero-Shot Compositional Learning](https://mlsanity.com/paper/1905.05908/) (2019)
- [DeepBach: a Steerable Model for Bach Chorales Generation](https://mlsanity.com/paper/1612.01010/) (2016)
- [Continuously Differentiable Exponential Linear Units](https://mlsanity.com/paper/1704.07483/) (2017)
- [Stratified Rule-Aware Network for Abstract Visual Reasoning](https://mlsanity.com/paper/2002.06838/) (2020)
- [Scale-Localized Abstract Reasoning](https://mlsanity.com/paper/2009.09405/) (2020)
- [Counterpoint by Convolution](https://mlsanity.com/paper/1903.07227/) (2019)
- [SpinalNet: Deep Neural Network with Gradual Input](https://mlsanity.com/paper/2007.03347/) (2020)
- [Norm-preserving Orthogonal Permutation Linear Unit Activation Functions (OPLU)](https://mlsanity.com/paper/1604.02313/) (2016)
- [A parameterized activation function for learning fuzzy logic operations in deep neural networks](https://mlsanity.com/paper/1708.08557/) (2017)

## Built on by (most cited first, 2 of 2)

- [Interpretable Classification via a Rule Network with Selective Logical Operators](https://mlsanity.com/paper/2408.11918/) (2024)
- [Kronecker Factorization Improves Efficiency and Interpretability of Sparse Autoencoders](https://mlsanity.com/paper/2505.22255/) (2025)

## Abstract (arXiv)

The choice of activation functions and their motivation is a long-standing issue within the neural network community. Neuronal representations within artificial neural networks are commonly understood as logits, representing the log-odds score of presence of features within the stimulus. We derive logit-space operators equivalent to probabilistic Boolean logic-gates AND, OR, and XNOR for independent probabilities. Such theories are important to formalize more complex dendritic operations in real neurons, and these operations can be used as activation functions within a neural network, introducing probabilistic Boolean-logic as the core operation of the neural network. Since these functions involve taking multiple exponents and logarithms, they are computationally expensive and not well suited to be directly used within neural networks. Consequently, we construct efficient approximations named ANDAIL (the AND operator Approximate for Independent Logits), ORAIL, and XNORAIL, which utilize only comparison and addition operations, have well-behaved gradients, and can be deployed as activation functions in neural networks. Like MaxOut, ANDAIL and ORAIL are generalizations of ReLU to two-dimensions. While our primary aim is to formalize dendritic computations within a logit-space probabilistic-Boolean framework, we deploy these new activation functions, both in isolation and in conjunction to demonstrate their effectiveness on a variety of tasks including image classification, transfer learning, abstract reasoning, and compositional zero-shot learning.