# GELU Activation Function in Deep Learning: A Comprehensive Mathematical Analysis and Performance

arXiv: https://arxiv.org/abs/2305.12073 · HTML: https://mlsanity.com/paper/2305.12073/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 35 references that are in the map.
- First posted on arXiv 20 May 2023.

## Builds on (most cited first, 22 of 22)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Gaussian Error Linear Units (GELUs)](https://mlsanity.com/paper/1606.08415/) (2016)
- [ViViT: A Video Vision Transformer](https://mlsanity.com/paper/2103.15691/) (2021)
- [Group Normalization](https://mlsanity.com/paper/1803.08494/) (2018)
- [MLP-Mixer: An all-MLP Architecture for Vision](https://mlsanity.com/paper/2105.01601/) (2021)
- [Focal Loss for Dense Object Detection](https://mlsanity.com/paper/1708.02002/) (2017)
- [Image Segmentation Using Deep Learning: A Survey](https://mlsanity.com/paper/2001.05566/) (2020)
- [The Medical Segmentation Decathlon](https://mlsanity.com/paper/2106.05735/) (2021)
- [Generative Adversarial Networks: An Overview](https://mlsanity.com/paper/1710.07035/) (2017)
- [CoAtNet: Marrying Convolution and Attention for All Data Sizes](https://mlsanity.com/paper/2106.04803/) (2021)
- [CSWin Transformer: A General Vision Transformer Backbone with Cross-Shaped Windows](https://mlsanity.com/paper/2107.00652/) (2021)
- [Activation Functions in Deep Learning: A Comprehensive Survey and Benchmark](https://mlsanity.com/paper/2109.14545/) (2021)
- [Lipschitz Continuity in Model-based Reinforcement Learning](https://mlsanity.com/paper/1804.07193/) (2018)
- [A survey on modern trainable activation functions](https://mlsanity.com/paper/2005.00817/) (2020)
- [When Machine Learning Meets Privacy: A Survey and Outlook](https://mlsanity.com/paper/2011.11819/) (2020)
- [Estimation with Uncertainty via Conditional Generative Adversarial Networks](https://mlsanity.com/paper/2007.00334/) (2020)

## Built on by (most cited first, 16 of 16)

- [Neural Circuit Diagrams: Robust Diagrams for the Communication, Implementation, and Analysis of Deep Learning Architectures](https://mlsanity.com/paper/2402.05424/) (2024)
- [CoNO: Complex Neural Operator for Continuous Dynamical Systems](https://mlsanity.com/paper/2310.02094/) (2023)
- [GBT-SAM: A Parameter-Efficient Depth-Aware Model for Generalizable Brain tumour Segmentation on mp-MRI](https://mlsanity.com/paper/2503.04325/) (2025)
- [Selective Structured State Space for Multispectral-fused Small Target Detection](https://mlsanity.com/paper/2505.14043/) (2025)
- [Text Adversarial Attacks with Dynamic Outputs](https://mlsanity.com/paper/2509.22393/) (2025)
- [Efficient Real-World Deblurring using Single Images: AIM 2025 Challenge Report](https://mlsanity.com/paper/2510.12788/) (2025)
- [MV-DETR: Multi-modality indoor object detection by Multi-View DEtecton TRansformers](https://mlsanity.com/paper/2408.06604/) (2024)
- [Whisper based Cross-Lingual Phoneme Recognition between Vietnamese and English](https://mlsanity.com/paper/2508.19270/) (2025)
- [Spatially Optimized Compact Deep Metric Learning Model for Similarity Search](https://mlsanity.com/paper/2404.06593/) (2024)
- [Zorro: A Flexible and Differentiable Parametric Family of Activation Functions That Extends ReLU and GELU](https://mlsanity.com/paper/2409.19239/) (2024)
- [SpectMamba: Integrating Frequency and State Space Models for Enhanced Medical Image Detection](https://mlsanity.com/paper/2509.01080/) (2025)
- [GZSL-MoE: Apprentissage G'en'eralis'e Z'ero-Shot bas'e sur le M'elange d'Experts pour la Segmentation S'emantique de Nuages de Points 3DAppliqu'e `a un Jeu de Donn'ees d'Environnement de Collaboration Humain-Robot](https://mlsanity.com/paper/2509.22708/) (2025)
- [Approximation with SiLU Networks: Constant Depth and Exponential Rates for Basic Operations](https://mlsanity.com/paper/2512.12132/) (2025)
- [Physics-Guided Sequence-Based Generative Framework for Acoustic Metamaterial Inverse Design](https://mlsanity.com/paper/2606.09266/) (2026)
- [Adaptive Interpolatory Curve Subdivision with Learned Local Angles](https://mlsanity.com/paper/2609.17566/) (2026)
- [On the Necessity of Attention-FFN Split in Vision Transformers](https://mlsanity.com/paper/2610.10303/) (2026)

## Abstract (arXiv)

Selecting the most suitable activation function is a critical factor in the effectiveness of deep learning models, as it influences their learning capacity, stability, and computational efficiency. In recent years, the Gaussian Error Linear Unit (GELU) activation function has emerged as a dominant method, surpassing traditional functions such as the Rectified Linear Unit (ReLU) in various applications. This study presents a rigorous mathematical investigation of the GELU activation function, exploring its differentiability, boundedness, stationarity, and smoothness properties in detail. Additionally, we conduct an extensive experimental comparison of the GELU function against a broad range of alternative activation functions, utilizing a residual convolutional network trained on the CIFAR-10, CIFAR-100, and STL-10 datasets as the empirical testbed. Our results demonstrate the superior performance of GELU compared to other activation functions, establishing its suitability for a wide range of deep learning applications. This comprehensive study contributes to a more profound understanding of the underlying mathematical properties of GELU and provides valuable insights for practitioners aiming to select activation functions that optimally align with their specific objectives and constraints in deep learning.