Exploring the Relationship: Transformative Adaptive Activation Functions in Comparison to Other Activation Functions
Not on the map. Why?
Abstract (arXiv)
Neural networks are the state-of-the-art approach for many tasks and the activation function is one of the main building blocks that allow such performance. Recently, a novel transformative adaptive activation function (TAAF) allowing for any vertical and horizontal translation and scaling was proposed. This work sets the TAAF into the context of other activation functions. It shows that the TAAFs generalize over 50 existing activation functions and utilize similar concepts as over 70 other activation functions, underscoring the versatility of TAAFs. This comprehensive exploration positions TAAFs as a promising and adaptable addition to neural networks.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 63 of its 174 references that are in the map. First posted on arXiv 14 Feb 2024.
Builds on
63 of 174 references.
References · 63
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
- 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
- 2014Going Deeper with Convolutionscited by 4,231
- 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,590
- 2018Graph Neural Networks: A Review of Methods and Applicationscited by 1,571
- 2017A Survey on Deep Learning in Medical Image Analysiscited by 1,182
- 2015Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)cited by 1,061
- 2016Geometric deep learning: going beyond Euclidean datacited by 1,042
- 2014Deep Learning in Neural Networks: An Overviewcited by 765
- 2019Self-supervised Visual Feature Learning with Deep Neural Networks: A Surveycited by 656
- 2017Self-Normalizing Neural Networkscited by 591
- 2019Deep Learning for 3D Point Clouds: A Surveycited by 557
38 more
- 2018Object Detection with Deep Learning: A Reviewcited by 524
- 2015Empirical Evaluation of Rectified Activations in Convolutional Networkcited by 515
- 2020Pre-trained Models for Natural Language Processing: A Surveycited by 504
- 2017A Brief Survey of Deep Reinforcement Learningcited by 466
- 2017Efficient Processing of Deep Neural Networks: A Tutorial and Surveycited by 427
- 2019Deep Learning for Image Super-resolution: A Surveycited by 368
- 2018Deep Learning in Spiking Neural Networkscited by 258
- 2019A Survey of the Recent Architectures of Deep Convolutional Neural Networkscited by 211
- 2013Maxout Networkscited by 186
- 2020ReActNet: Towards Precise Binary Neural Network with Generalized Activation Functionscited by 183
- 2018Deep learning in agriculture: A surveycited by 160
- 2018Deep Learning for Single Image Super-Resolution: A Brief Reviewcited by 158
- 2021Activation Functions in Deep Learning: A Comprehensive Survey and Benchmarkcited by 153
- 2018An overview of deep learning in medical imaging focusing on MRIcited by 140
- 2019Deep Learning on Image Denoising: An overviewcited by 128
- 2019A Gentle Introduction to Deep Learning for Graphscited by 107
- 2017Deep Learning for Medical Image Processing: Overview, Challenges and Futurecited by 104
- 2017Medical Image Analysis using Convolutional Neural Networks: A Reviewcited by 99
- 2020A survey on modern trainable activation functionscited by 89
- 2017Continuously Differentiable Exponential Linear Unitscited by 73
- 2016Deep Learning in Bioinformaticscited by 48
- 2017Deep Learning for Computational Chemistrycited by 38
- 2014Spatially-sparse convolutional neural networkscited by 33
- 2019Learning Activation Functions: A new paradigm for understanding Neural Networkscited by 33
- 2016Parametric Exponential Linear Unit for Deep Convolutional Neural Networkscited by 32
- 2020Deep Learning Based Brain Tumor Segmentation: A Surveycited by 30
- 2015Deep Learning with S-shaped Rectified Linear Activation Unitscited by 26
- 2021Parameterizing Activation Functions for Adversarial Robustnesscited by 22
- 2018Learning Combinations of Activation Functionscited by 20
- 2020Improving Accuracy of Binary Neural Networks using Unbalanced Activation Distributioncited by 15
- 2021Truly Sparse Neural Networks at Scalecited by 15
- 2018Adaptive Blending Units: Trainable Activation Functions for Deep Neural Networkscited by 13
- 2016Revise Saturated Activation Functionscited by 11
- 2018PLU: The Piecewise Linear Unit Activation Functioncited by 11
- 2019A simple and efficient architecture for trainable activation functionscited by 10
- 2020AReLU: Attention-based Rectified Linear Unitcited by 9
- 2019ACTNET: end-to-end learning of feature activations and multi-stream aggregation for effective instance image retrievalcited by 7
- 2016Improving Deep Neural Network with Multiple Parametric Exponential Linear Unitscited by 6
50 most cited of 63 All 63 (JSON)
Built on by
Citing papers per year: 2025: 1.
1 paper.