mlsanityData to 7 Oct 2026

Increasing biases can be more efficient than increasing weights

arXiv 2301.00924 ↗First posted 3 Jan 2023cs.NE cs.LGcited by 4

Not on the map. Why?

Abstract (arXiv)

We introduce a novel computational unit for neural networks that features multiple biases, challenging the traditional perceptron structure. This unit emphasizes the importance of preserving uncorrupted information as it is passed from one unit to the next, applying activation functions later in the process with specialized biases for each unit. Through both empirical and theoretical analyses, we show that by focusing on increasing biases rather than weights, there is potential for significant enhancement in a neural network model's performance. This approach offers an alternative perspective on optimizing information flow within neural networks. See source code at https://github.com/CuriosAI/dac-dev.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 22 of its 51 references that are in the map. First posted on arXiv 3 Jan 2023.

Builds on

22 of 51 references.

References · 22

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
  5. 2022Generative Adversarial Networkscited by 5,603
  6. 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,773
  7. 2022A ConvNet for the 2020scited by 3,545
  8. 2016Xception: Deep Learning with Depthwise Separable Convolutionscited by 2,320
  9. 2016Identity Mappings in Deep Residual Networkscited by 1,960
  10. 2015Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)cited by 1,061
  11. 2019Green AIcited by 620
  12. 2013Maxout Networkscited by 186
10 more
  1. 2016Understanding and Improving Convolutional Neural Networks via Concatenated Rectified Linear Unitscited by 86
  2. 2014Learning Activation Functions to Improve Deep Neural Networkscited by 62
  3. 2019Accelerating Self-Play Learning in Gocited by 60
  4. 2017ISIC 2017 - Skin Lesion Analysis Towards Melanoma Detectioncited by 34
  5. 2017Activation Ensembles for Deep Neural Networkscited by 5
  6. 2018SAI, a Sensible Artificial Intelligence that plays Gocited by 5
  7. 2016Multi-Bias Non-linear Activation in Deep Neural Networkscited by 4
  8. 2019SAI: a Sensible Artificial Intelligence that plays with handicap and targets high scores in 9x9 Go (extended version)cited by 2
  9. 2022Score vs. Winrate in Score-Based Games: which Reward for Reinforcement Learning?cited by 2
  10. 2021Curious Explorer: a provable exploration strategy in Policy Learningcited by 1

Built on by

Cited by year

Citing papers per year: 2023: 1; 2024: 3.

4 papers.

  1. 2024Volvo Discovery Challenge at ECML-PKDD 2024cited by 2
  2. 2023GloNets: Globally Connected Neural Networkscited by 1
  3. 2024A Systematization of the Wagner Framework: Graph Theory Conjectures and Reinforcement Learningcited by 1
  4. 2024Achieving Predictive Precision: Leveraging LSTM and Pseudo Labeling for Volvo's Discovery Challenge at ECML-PKDD 2024cited by 0