# Optimizing Performance of Feedforward and Convolutional Neural Networks through Dynamic Activation Functions

arXiv: https://arxiv.org/abs/2308.05724 · HTML: https://mlsanity.com/paper/2308.05724/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 16 of its 98 references that are in the map.
- First posted on arXiv 10 Aug 2023.

## Builds on (most cited first, 16 of 16)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Convex optimization](https://mlsanity.com/paper/2106.01946/) (2021)
- [Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling](https://mlsanity.com/paper/1412.3555/) (2014)
- [Visualizing and Understanding Convolutional Networks](https://mlsanity.com/paper/1311.2901/) (2013)
- [Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting](https://mlsanity.com/paper/1506.04214/) (2015)
- [Generalized Cross Entropy Loss for Training Deep Neural Networks with Noisy Labels](https://mlsanity.com/paper/1805.07836/) (2018)
- [Long-term Recurrent Convolutional Networks for Visual Recognition and Description](https://mlsanity.com/paper/1411.4389/) (2014)
- [Activation Functions: Comparison of trends in Practice and Research for Deep Learning](https://mlsanity.com/paper/1811.03378/) (2018)
- [Learning Activation Functions to Improve Deep Neural Networks](https://mlsanity.com/paper/1412.6830/) (2014)
- [Collapse of Deep and Narrow Neural Nets](https://mlsanity.com/paper/1808.04947/) (2018)
- [PLU: The Piecewise Linear Unit Activation Function](https://mlsanity.com/paper/1809.09534/) (2018)
- [Enhanced Higgs to τ^+τ^- Searches with Deep Learning](https://mlsanity.com/paper/1410.3469/) (2014)
- [Human Action Recognition and Assessment via Deep Neural Network Self-Organization](https://mlsanity.com/paper/2001.05837/) (2020)
- [Optimal Input Gain: All You Need to Supercharge a Feed-Forward Neural Network](https://mlsanity.com/paper/2303.17732/) (2023)

## Abstract (arXiv)

Deep learning training training algorithms are a huge success in recent years in many fields including speech, text,image video etc. Deeper and deeper layers are proposed with huge success with resnet structures having around 152 layers. Shallow convolution neural networks(CNN's) are still an active research, where some phenomena are still unexplained. Activation functions used in the network are of utmost importance, as they provide non linearity to the networks. Relu's are the most commonly used activation function.We show a complex piece-wise linear(PWL) activation in the hidden layer. We show that these PWL activations work much better than relu activations in our networks for convolution neural networks and multilayer perceptrons. Result comparison in PyTorch for shallow and deep CNNs are given to further strengthen our case.