mlsanityData to 7 Oct 2026

Orthogonal-Pad'e Activation Functions: Trainable Activation functions for smooth and faster convergence in deep networks

arXiv 2106.09693 ↗First posted 17 Jun 2021cs.NE cs.AI cs.CVcited by 4

Not on the map. Why?

Abstract (arXiv)

We have proposed orthogonal-Pad'e activation functions, which are trainable activation functions and show that they have faster learning capability and improves the accuracy in standard deep learning datasets and models. Based on our experiments, we have found two best candidates out of six orthogonal-Pad'e activations, which we call safe Hermite-Pade (HP) activation functions, namely HP-1 and HP-2. When compared to ReLU, HP-1 and HP-2 has an increment in top-1 accuracy by 5.06% and 4.63% respectively in PreActResNet-34, by 3.02% and 2.75% respectively in MobileNet V2 model on CIFAR100 dataset while on CIFAR10 dataset top-1 accuracy increases by 2.02% and 1.78% respectively in PreActResNet-34, by 2.24% and 2.06% respectively in LeNet, by 2.15% and 2.03% respectively in Efficientnet B0.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 20 of its 38 references that are in the map. First posted on arXiv 17 Jun 2021.

Builds on

20 of 38 references.

References · 20

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  4. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  5. 2016Densely Connected Convolutional Networkscited by 6,691
  6. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
  7. 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,773
  8. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,283
  9. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,590
  10. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  11. 2015SSD: Single Shot MultiBox Detectorcited by 3,125
  12. 2016Wide Residual Networkscited by 3,057
8 more
  1. 2016Identity Mappings in Deep Residual Networkscited by 1,960
  2. 2018ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Designcited by 1,132
  3. 2015Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)cited by 1,061
  4. 2017Deep Layer Aggregationcited by 436
  5. 2019Universal Approximation with Deep Narrow Networkscited by 198
  6. 2019Pad'e Activation Units: End-to-end Learning of Flexible Activation Functions in Deep Networkscited by 65
  7. 2020TanhSoft -- a family of activation functions combining Tanh and Softpluscited by 2
  8. 2020EIS -- a family of activation functions combining Exponential, ISRU, and Softpluscited by 1

Built on by

Cited by year

Citing papers per year: 2021: 3; 2022: 0; 2023: 0; 2024: 0; 2025: 1.

4 papers.

  1. 2021SMU: smooth activation function for deep networks using smoothing maximum techniquecited by 7
  2. 2021ErfAct and Pserf: Non-monotonic Smooth Trainable Activation Functionscited by 5
  3. 2021SAU: Smooth activation function using convolution with approximate identitiescited by 1
  4. 2025Polynomial, trigonometric, and tropical activationscited by 0