mlsanityData to 7 Oct 2026

SMU: smooth activation function for deep networks using smoothing maximum technique

arXiv 2111.04682 ↗First posted 8 Nov 2021cs.LG cs.AI cs.CVcited by 7

Not on the map. Why?

Abstract (arXiv)

Deep learning researchers have a keen interest in proposing two new novel activation functions which can boost network performance. A good choice of activation function can have significant consequences in improving network performance. A handcrafted activation is the most common choice in neural network models. ReLU is the most common choice in the deep learning community due to its simplicity though ReLU has some serious drawbacks. In this paper, we have proposed a new novel activation function based on approximation of known activation functions like Leaky ReLU, and we call this function Smooth Maximum Unit (SMU). Replacing ReLU by SMU, we have got 6.22% improvement in the CIFAR100 dataset with the ShuffleNet V2 model.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Builds on 21 of its 35 references that are in the map. First posted on arXiv 8 Nov 2021.

Builds on

21 of 35 references.

References · 21

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  5. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  6. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  7. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  8. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
  9. 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,773
  10. 2016The Cityscapes Dataset for Semantic Urban Scene Understandingcited by 4,427
  11. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,590
  12. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
9 more
  1. 2015SSD: Single Shot MultiBox Detectorcited by 3,125
  2. 2016Identity Mappings in Deep Residual Networkscited by 1,960
  3. 2018ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Designcited by 1,132
  4. 2015Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)cited by 1,061
  5. 2015Empirical Evaluation of Rectified Activations in Convolutional Networkcited by 515
  6. 2019Universal Approximation with Deep Narrow Networkscited by 198
  7. 2019Pad'e Activation Units: End-to-end Learning of Flexible Activation Functions in Deep Networkscited by 65
  8. 2021ErfAct and Pserf: Non-monotonic Smooth Trainable Activation Functionscited by 5
  9. 2021Orthogonal-Pad'e Activation Functions: Trainable Activation functions for smooth and faster convergence in deep networkscited by 4

Built on by

Cited by year

Citing papers per year: 2022: 1; 2023: 2; 2024: 3; 2025: 1.

7 papers.

  1. 2023Efficient Federated Learning via Local Adaptive Amended Optimizer with Linear Speedupcited by 23
  2. 2024emg2pose: A Large and Diverse Benchmark for Surface Electromyographic Hand Pose Estimationcited by 20
  3. 2022The Outcome of the 2022 Landslide4Sense Competition: Advanced Landslide Detection from Multi-Source Satellite Imagerycited by 15
  4. 2024Comet: A Communication-efficient and Performant Approximation for Private Transformer Inferencecited by 7
  5. 2023Interaction-Aware Trajectory Planning for Autonomous Vehicles with Analytic Integration of Neural Networks into Model Predictive Controlcited by 5
  6. 2024Associative memory and dead neuronscited by 1
  7. 2025Regularizing Differentiable Architecture Search with Smooth Activationcited by 1