# Why do networks have inhibitory/negative connections?

arXiv: https://arxiv.org/abs/2208.03211 · HTML: https://mlsanity.com/paper/2208.03211/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 5 of its 40 references that are in the map.
- First posted on arXiv 5 Aug 2022.

## Builds on (most cited first, 5 of 5)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Input Convex Neural Networks](https://mlsanity.com/paper/1609.07152/) (2016)
- [Universality of Deep Convolutional Neural Networks](https://mlsanity.com/paper/1805.10769/) (2018)
- [Why and When Can Deep -- but Not Shallow -- Networks Avoid the Curse of Dimensionality: a Review](https://mlsanity.com/paper/1611.00740/) (2016)
- [Convolutional Rectifier Networks as Generalized Tensor Decompositions](https://mlsanity.com/paper/1603.00162/) (2016)

## Built on by (most cited first, 4 of 4)

- [A Fully-Configurable Open-Source Software-Defined Digital Quantized Spiking Neural Core Architecture](https://mlsanity.com/paper/2404.02248/) (2024)
- [Polarity is all you need to learn and transfer faster](https://mlsanity.com/paper/2303.17589/) (2023)
- [Find A Winning Sign: Sign Is All We Need to Win the Lottery](https://mlsanity.com/paper/2504.05357/) (2025)
- [Understanding Inhibition Through Maximally Tense Images](https://mlsanity.com/paper/2406.05598/) (2024)

## Abstract (arXiv)

Why do brains have inhibitory connections? Why do deep networks have negative weights? We propose an answer from the perspective of representation capacity. We believe representing functions is the primary role of both (i) the brain in natural intelligence, and (ii) deep networks in artificial intelligence. Our answer to why there are inhibitory/negative weights is: to learn more functions. We prove that, in the absence of negative weights, neural networks with non-decreasing activation functions are not universal approximators. While this may be an intuitive result to some, to the best of our knowledge, there is no formal theory, in either machine learning or neuroscience, that demonstrates why negative weights are crucial in the context of representation capacity. Further, we provide insights on the geometric properties of the representation space that non-negative deep networks cannot represent. We expect these insights will yield a deeper understanding of more sophisticated inductive priors imposed on the distribution of weights that lead to more efficient biological and machine learning.