# Increasing biases can be more efficient than increasing weights

arXiv: https://arxiv.org/abs/2301.00924 · HTML: https://mlsanity.com/paper/2301.00924/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 51 references that are in the map.
- First posted on arXiv 3 Jan 2023.

## Builds on (most cited first, 22 of 22)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [Generative Adversarial Networks](https://mlsanity.com/paper/2203.00667/) (2022)
- [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018)
- [A ConvNet for the 2020s](https://mlsanity.com/paper/2201.03545/) (2022)
- [Xception: Deep Learning with Depthwise Separable Convolutions](https://mlsanity.com/paper/1610.02357/) (2016)
- [Identity Mappings in Deep Residual Networks](https://mlsanity.com/paper/1603.05027/) (2016)
- [Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)](https://mlsanity.com/paper/1511.07289/) (2015)
- [Green AI](https://mlsanity.com/paper/1907.10597/) (2019)
- [Maxout Networks](https://mlsanity.com/paper/1302.4389/) (2013)
- [Understanding and Improving Convolutional Neural Networks via Concatenated Rectified Linear Units](https://mlsanity.com/paper/1603.05201/) (2016)
- [Accelerating Self-Play Learning in Go](https://mlsanity.com/paper/1902.10565/) (2019)
- [Learning Activation Functions to Improve Deep Neural Networks](https://mlsanity.com/paper/1412.6830/) (2014)
- [ISIC 2017 - Skin Lesion Analysis Towards Melanoma Detection](https://mlsanity.com/paper/1703.00523/) (2017)
- [Activation Ensembles for Deep Neural Networks](https://mlsanity.com/paper/1702.07790/) (2017)
- [SAI, a Sensible Artificial Intelligence that plays Go](https://mlsanity.com/paper/1809.03928/) (2018)
- [Multi-Bias Non-linear Activation in Deep Neural Networks](https://mlsanity.com/paper/1604.00676/) (2016)
- [SAI: a Sensible Artificial Intelligence that plays with handicap and targets high scores in 9x9 Go (extended version)](https://mlsanity.com/paper/1905.10863/) (2019)
- [Score vs. Winrate in Score-Based Games: which Reward for Reinforcement Learning?](https://mlsanity.com/paper/2201.13176/) (2022)
- [Curious Explorer: a provable exploration strategy in Policy Learning](https://mlsanity.com/paper/2106.15503/) (2021)

## Built on by (most cited first, 4 of 4)

- [Volvo Discovery Challenge at ECML-PKDD 2024](https://mlsanity.com/paper/2409.11446/) (2024)
- [GloNets: Globally Connected Neural Networks](https://mlsanity.com/paper/2311.15947/) (2023)
- [A Systematization of the Wagner Framework: Graph Theory Conjectures and Reinforcement Learning](https://mlsanity.com/paper/2406.12667/) (2024)
- [Achieving Predictive Precision: Leveraging LSTM and Pseudo Labeling for Volvo's Discovery Challenge at ECML-PKDD 2024](https://mlsanity.com/paper/2409.13877/) (2024)

## Abstract (arXiv)

We introduce a novel computational unit for neural networks that features multiple biases, challenging the traditional perceptron structure. This unit emphasizes the importance of preserving uncorrupted information as it is passed from one unit to the next, applying activation functions later in the process with specialized biases for each unit. Through both empirical and theoretical analyses, we show that by focusing on increasing biases rather than weights, there is potential for significant enhancement in a neural network model's performance. This approach offers an alternative perspective on optimizing information flow within neural networks. See source code at https://github.com/CuriosAI/dac-dev.