mlsanityData to 14 Oct 2026

A Non-monotonic Smooth Activation Function

arXiv 2310.10126 ↗First posted 16 Oct 2023cs.LG cs.AIcited by 3

Not on the map. Why?

Abstract (arXiv)

Activation functions are crucial in deep learning models since they introduce non-linearity into the networks, allowing them to learn from errors and make adjustments, which is essential for learning complex patterns. The essential purpose of activation functions is to transform unprocessed input signals into significant output activations, promoting information transmission throughout the neural network. In this study, we propose a new activation function called Sqish, which is a non-monotonic and smooth function and an alternative to existing ones. We showed its superiority in classification, object detection, segmentation tasks, and adversarial robustness experiments. We got an 8.21% improvement over ReLU on the CIFAR100 dataset with the ShuffleNet V2 model in the FGSM adversarial attack. We also got a 5.87% improvement over ReLU on image classification on the CIFAR100 dataset with the ShuffleNet V2 model.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 27 of its 45 references that are in the map. First posted on arXiv 16 Oct 2023.

Builds on

27 of 45 references.

References · 27

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  4. 2020Language Models are Few-Shot Learnerscited by 32,419
  5. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  6. 2014Explaining and Harnessing Adversarial Examplescited by 7,364
  7. 2016Densely Connected Convolutional Networkscited by 6,720
  8. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,442
  9. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,475
  10. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,183
  11. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 5,045
  12. 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,795
15 more
  1. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,349
  2. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,611
  3. 2016Gaussian Error Linear Units (GELUs)cited by 3,502
  4. 2015SSD: Single Shot MultiBox Detectorcited by 3,130
  5. 2016Aggregated Residual Transformations for Deep Neural Networkscited by 2,546
  6. 2016Xception: Deep Learning with Depthwise Separable Convolutionscited by 2,332
  7. 2016Identity Mappings in Deep Residual Networkscited by 1,965
  8. 20163D U-Net: Learning Dense Volumetric Segmentation from Sparse Annotationcited by 1,476
  9. 2018ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Designcited by 1,136
  10. 2015Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)cited by 1,070
  11. 2015Empirical Evaluation of Rectified Activations in Convolutional Networkcited by 519
  12. 2018Identifying the Best Machine Learning Algorithms for Brain Tumor Segmentation, Progression Assessment, and Overall Survival Prediction in the BRATS Challengecited by 511
  13. 2019Universal Approximation with Deep Narrow Networkscited by 200
  14. 2020MosMedData: Chest CT Scans With COVID-19 Related Findings Datasetcited by 104
  15. 2019Pad'e Activation Units: End-to-end Learning of Flexible Activation Functions in Deep Networkscited by 67

Built on by

Cited by year

Citing papers per year: 2024: 1; 2025: 2.

3 papers.

  1. 2024Zorro: A Flexible and Differentiable Parametric Family of Activation Functions That Extends ReLU and GELUcited by 0
  2. 2025Robustness as Architecture: Designing IQA Models to Withstand Adversarial Perturbationscited by 0
  3. 2025Automatic Contouring of Spinal Vertebrae on X-Ray using a Novel Sandwich U-Net Architecturecited by 0