# Neural Networks with Smooth Adaptive Activation Functions for Regression

arXiv: https://arxiv.org/abs/1608.06557 · HTML: https://mlsanity.com/paper/1608.06557/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 Aug 2016.

## Built on by (most cited first, 3 of 3)

- [ENN: A Neural Network with DCT Adaptive Activation Functions](https://mlsanity.com/paper/2307.00673/) (2023)
- [TeLU Activation Function for Fast and Stable Deep Learning](https://mlsanity.com/paper/2412.20269/) (2024)
- [Robust Deep Network Learning of Nonlinear Regression Tasks by Parametric Leaky Exponential Linear Units (LELUs) and a Diffusion Metric](https://mlsanity.com/paper/2507.06765/) (2025)

## Abstract (arXiv)

In Neural Networks (NN), Adaptive Activation Functions (AAF) have parameters that control the shapes of activation functions. These parameters are trained along with other parameters in the NN. AAFs have improved performance of Neural Networks (NN) in multiple classification tasks. In this paper, we propose and apply AAFs on feedforward NNs for regression tasks. We argue that applying AAFs in the regression (second-to-last) layer of a NN can significantly decrease the bias of the regression NN. However, using existing AAFs may lead to overfitting. To address this problem, we propose a Smooth Adaptive Activation Function (SAAF) with piecewise polynomial form which can approximate any continuous function to arbitrary degree of error. NNs with SAAFs can avoid overfitting by simply regularizing the parameters. In particular, an NN with SAAFs is Lipschitz continuous given a bounded magnitude of the NN parameters. We prove an upper-bound for model complexity in terms of fat-shattering dimension for any Lipschitz continuous regression model. Thus, regularizing the parameters in NNs with SAAFs avoids overfitting. We empirically evaluated NNs with SAAFs and achieved state-of-the-art results on multiple regression datasets.