# Deep Learning with S-shaped Rectified Linear Activation Units

arXiv: https://arxiv.org/abs/1512.07030 · HTML: https://mlsanity.com/paper/1512.07030/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 22 Dec 2015.

## Built on by (most cited first, 26 of 26)

- [Activation Functions in Deep Learning: A Comprehensive Survey and Benchmark](https://mlsanity.com/paper/2109.14545/) (2021)
- [How important are activation functions in regression and classification? A survey, performance comparison, and future directions](https://mlsanity.com/paper/2209.02681/) (2022)
- [Improving Lipschitz-Constrained Neural Networks by Learning Activation Functions](https://mlsanity.com/paper/2210.16222/) (2022)
- [Keep the Gradients Flowing: Using Gradient Flow to Study Sparse Network Optimization](https://mlsanity.com/paper/2102.01670/) (2021)
- [Truly Sparse Neural Networks at Scale](https://mlsanity.com/paper/2102.01732/) (2021)
- [HOSC: A Periodic Activation Function for Preserving Sharp Features in Implicit Neural Representations](https://mlsanity.com/paper/2401.10967/) (2024)
- [Advancing Constrained Monotonic Neural Networks: Achieving Universal Approximation Beyond Bounded Activations](https://mlsanity.com/paper/2505.02537/) (2025)
- [Piecewise Linear Neural Networks and Deep Learning](https://mlsanity.com/paper/2206.09149/) (2022)
- [Towards NeuroAI: Introducing Neuronal Diversity into Artificial Neural Networks](https://mlsanity.com/paper/2301.09245/) (2023)
- [Neural Characteristic Activation Analysis and Geometric Parameterization for ReLU Networks](https://mlsanity.com/paper/2305.15912/) (2023)
- [Why KDAC? A general activation function for knowledge discovery](https://mlsanity.com/paper/2111.13858/) (2021)
- [ENN: A Neural Network with DCT Adaptive Activation Functions](https://mlsanity.com/paper/2307.00673/) (2023)
- [Comparison of different convolutional neural network activation functions and methods for building ensembles](https://mlsanity.com/paper/2103.15898/) (2021)
- [Real World Large Scale Recommendation Systems Reproducibility and Smooth Activations](https://mlsanity.com/paper/2202.06499/) (2022)
- [Trainable Compound Activation Functions for Machine Learning](https://mlsanity.com/paper/2204.12920/) (2022)
- [Empirical study of the modulus as activation function in computer vision applications](https://mlsanity.com/paper/2301.05993/) (2023)
- [Deep ensembles based on Stochastic Activation Selection for Polyp Segmentation](https://mlsanity.com/paper/2104.00850/) (2021)
- [Deep ensembles in bioimage segmentation](https://mlsanity.com/paper/2112.12955/) (2021)
- [Learning Activation Functions for Sparse Neural Networks](https://mlsanity.com/paper/2305.10964/) (2023)
- [Exploring the Relationship: Transformative Adaptive Activation Functions in Comparison to Other Activation Functions](https://mlsanity.com/paper/2402.09249/) (2024)
- [Activation Function Optimization Scheme for Image Classification](https://mlsanity.com/paper/2409.04915/) (2024)
- [Optimizing Neural Networks with Learnable Non-Linear Activation Functions via Lookup-Based FPGA Acceleration](https://mlsanity.com/paper/2508.17069/) (2025)
- [Unification of popular artificial neural network activation functions](https://mlsanity.com/paper/2302.11007/) (2023)
- [Activations Through Extensions: A Framework To Boost Performance Of Neural Networks](https://mlsanity.com/paper/2408.03599/) (2024)
- [Deep Network Trainability via Persistent Subspace Orthogonality](https://mlsanity.com/paper/2508.02882/) (2025)
- [dynActivation: A Trainable Activation Family for Adaptive Nonlinearity](https://mlsanity.com/paper/2603.22154/) (2026)

## Abstract (arXiv)

Rectified linear activation units are important components for state-of-the-art deep convolutional networks. In this paper, we propose a novel S-shaped rectified linear activation unit (SReLU) to learn both convex and non-convex functions, imitating the multiple function forms given by the two fundamental laws, namely the Webner-Fechner law and the Stevens law, in psychophysics and neural sciences. Specifically, SReLU consists of three piecewise linear functions, which are formulated by four learnable parameters. The SReLU is learned jointly with the training of the whole deep network through back propagation. During the training phase, to initialize SReLU in different layers, we propose a "freezing" method to degenerate SReLU into a predefined leaky rectified linear unit in the initial several training epochs and then adaptively learn the good initial values. SReLU can be universally used in the existing deep networks with negligible additional parameters and computation cost. Experiments with two popular CNN architectures, Network in Network and GoogLeNet on scale-various benchmarks including CIFAR10, CIFAR100, MNIST and ImageNet demonstrate that SReLU achieves remarkable improvement compared to other activation functions.