mlsanityData to 14 Oct 2026

SAU: Smooth activation function using convolution with approximate identities

arXiv 2109.13210 ↗First posted 27 Sep 2021cs.LG cs.AI cs.CVcited by 1

Not on the map. Why?

Abstract (arXiv)

Well-known activation functions like ReLU or Leaky ReLU are non-differentiable at the origin. Over the years, many smooth approximations of ReLU have been proposed using various smoothing techniques. We propose new smooth approximations of a non-differentiable activation function by convolving it with approximate identities. In particular, we present smooth approximations of Leaky ReLU and show that they outperform several well-known activation functions in various datasets and models. We call this function Smooth Activation Unit (SAU). Replacing ReLU by SAU, we get 5.12% improvement with ShuffleNet V2 (2.0x) model on CIFAR100 dataset.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 31 of its 54 references that are in the map. First posted on arXiv 27 Sep 2021.

Builds on

31 of 54 references.

References · 31

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  4. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  5. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,713
  6. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  7. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 7,005
  8. 2016Densely Connected Convolutional Networkscited by 6,720
  9. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,442
  10. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,475
  11. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,183
  12. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 5,045
19 more
  1. 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,795
  2. 2017Squeeze-and-Excitation Networkscited by 4,789
  3. 2016The Cityscapes Dataset for Semantic Urban Scene Understandingcited by 4,444
  4. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,349
  5. 2017MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applicationscited by 3,845
  6. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,611
  7. 2016Gaussian Error Linear Units (GELUs)cited by 3,502
  8. 2015SSD: Single Shot MultiBox Detectorcited by 3,130
  9. 2016Wide Residual Networkscited by 3,070
  10. 2016Aggregated Residual Transformations for Deep Neural Networkscited by 2,546
  11. 2016Xception: Deep Learning with Depthwise Separable Convolutionscited by 2,332
  12. 2016Identity Mappings in Deep Residual Networkscited by 1,965
  13. 2018ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Designcited by 1,136
  14. 2015Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)cited by 1,070
  15. 2019Universal Approximation with Deep Narrow Networkscited by 200
  16. 2019Pad'e Activation Units: End-to-end Learning of Flexible Activation Functions in Deep Networkscited by 67
  17. 2020Activate or Not: Learning Customized Activationcited by 22
  18. 2021ErfAct and Pserf: Non-monotonic Smooth Trainable Activation Functionscited by 5
  19. 2021Orthogonal-Pad'e Activation Functions: Trainable Activation functions for smooth and faster convergence in deep networkscited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Geometric Monomial (GEM): a family of rational 2N-differentiable activation functionscited by 0