mlsanityData to 14 Oct 2026

ErfAct and Pserf: Non-monotonic Smooth Trainable Activation Functions

arXiv 2109.04386 ↗First posted 9 Sep 2021cs.NE cs.AI cs.CVcited by 5

Not on the map. Why?

Abstract (arXiv)

An activation function is a crucial component of a neural network that introduces non-linearity in the network. The state-of-the-art performance of a neural network depends also on the perfect choice of an activation function. We propose two novel non-monotonic smooth trainable activation functions, called ErfAct and Pserf. Experiments suggest that the proposed functions improve the network performance significantly compared to the widely used activations like ReLU, Swish, and Mish. Replacing ReLU by ErfAct and Pserf, we have 5.68% and 5.42% improvement for top-1 accuracy on Shufflenet V2 (2.0x) network in CIFAR100 dataset, 2.11% and 1.96% improvement for top-1 accuracy on Shufflenet V2 (2.0x) network in CIFAR10 dataset, 1.0%, and 1.0% improvement on mean average precision (mAP) on SSD300 model in Pascal VOC dataset.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 37 of its 64 references that are in the map. First posted on arXiv 9 Sep 2021.

Builds on

37 of 64 references.

References · 37

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  4. 2020Language Models are Few-Shot Learnerscited by 32,419
  5. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,713
  6. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  7. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  8. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 7,005
  9. 2016Densely Connected Convolutional Networkscited by 6,720
  10. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,442
  11. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,475
  12. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,183
25 more
  1. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 5,045
  2. 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,795
  3. 2016The Cityscapes Dataset for Semantic Urban Scene Understandingcited by 4,444
  4. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,349
  5. 2014Going Deeper with Convolutionscited by 4,241
  6. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,611
  7. 2016Gaussian Error Linear Units (GELUs)cited by 3,502
  8. 2015SSD: Single Shot MultiBox Detectorcited by 3,130
  9. 2016Wide Residual Networkscited by 3,070
  10. 2016Aggregated Residual Transformations for Deep Neural Networkscited by 2,546
  11. 2016Xception: Deep Learning with Depthwise Separable Convolutionscited by 2,332
  12. 2016Identity Mappings in Deep Residual Networkscited by 1,965
  13. 2020YOLOv4: Optimal Speed and Accuracy of Object Detectioncited by 1,630
  14. 2018ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Designcited by 1,136
  15. 2015Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)cited by 1,070
  16. 2016SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model sizecited by 1,060
  17. 2017Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learningcited by 950
  18. 2013Network In Networkcited by 605
  19. 2017Deep Layer Aggregationcited by 436
  20. 2019Universal Approximation with Deep Narrow Networkscited by 200
  21. 2019Pad'e Activation Units: End-to-end Learning of Flexible Activation Functions in Deep Networkscited by 67
  22. 2016Lets keep it simple, Using simple architectures to outperform deeper and more complex architecturescited by 25
  23. 2020Activate or Not: Learning Customized Activationcited by 22
  24. 2021SERF: Towards better training of deep neural networks using log-Softplus ERror activation Functioncited by 10
  25. 2021Orthogonal-Pad'e Activation Functions: Trainable Activation functions for smooth and faster convergence in deep networkscited by 4

Built on by

Cited by year

Citing papers per year: 2021: 2; 2022: 0; 2023: 1; 2024: 2.

5 papers.

  1. 2021SMU: smooth activation function for deep networks using smoothing maximum techniquecited by 7
  2. 2024Kolmogorov-Arnold Networks in Low-Data Regimes: A Comparative Study with Multilayer Perceptronscited by 5
  3. 2023ErfReLU: Adaptive Activation Function for Deep Neural Networkcited by 3
  4. 2021SAU: Smooth activation function using convolution with approximate identitiescited by 1
  5. 2024Swish-T : Enhancing Swish Activation with Tanh Bias for Improved Neural Network Performancecited by 0