mlsanityData to 14 Oct 2026

Nonlinearity Enhanced Adaptive Activation Functions

arXiv 2403.19896 ↗First posted 29 Mar 2024cs.LG cs.CV cs.NEcited by 0

Not on the map. Why?

Abstract (arXiv)

A general procedure for introducing parametric, learned, nonlinearity into activation functions is found to enhance the accuracy of representative neural networks without requiring significant additional computational resources. Examples are given based on the standard rectified linear unit (ReLU) as well as several other frequently employed activation functions. The associated accuracy improvement is quantified both in the context of the MNIST digit data set and a convolutional neural network (CNN) benchmark example.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 16 of its 36 references that are in the map. First posted on arXiv 29 Mar 2024.

Builds on

16 of 36 references.

References · 16

  1. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,611
  2. 2016Gaussian Error Linear Units (GELUs)cited by 3,502
  3. 2015Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)cited by 1,070
  4. 2018A high-bias, low-variance introduction to Machine Learning for physicistscited by 108
  5. 2020Neural Networks Fail to Learn Periodic Functions and How to Fix Itcited by 100
  6. 2020A survey on modern trainable activation functionscited by 89
  7. 2014Learning Activation Functions to Improve Deep Neural Networkscited by 63
  8. 2022How important are activation functions in regression and classification? A survey, performance comparison, and future directionscited by 29
  9. 2018Learning Combinations of Activation Functionscited by 20
  10. 2018Adaptive Blending Units: Trainable Activation Functions for Deep Neural Networkscited by 13
  11. 2021Activation Functions in Artificial Neural Networks: A Systematic Overviewcited by 13
  12. 2018E-swish: Adjusting Activations to Different Network Depthscited by 6
4 more
  1. 2022A survey on recently proposed activation functions for Deep Learningcited by 6
  2. 2018MaxGain: Regularisation of Neural Networks by Constraining Activation Magnitudescited by 3
  3. 2021Formalising the Use of the Activation Function in Neural Inferencecited by 1
  4. 2023Neural Network Characterization and Entropy Regulated Data Balancing through Principal Component Analysiscited by 1

Built on by

Not cited yet.