Data to 7 Oct 2026

Searching for Activation Functions

arXiv 1710.05941 ↗First posted 16 Oct 2017cs.NE cs.CV cs.LGcited by 158

Not on the map. Why?

Abstract (arXiv)

The choice of activation functions in deep networks has a significant effect on the training dynamics and task performance. Currently, the most successful and widely-used activation function is the Rectified Linear Unit (ReLU). Although various hand-designed alternatives to ReLU have been proposed, none have managed to replace it due to inconsistent gains. In this work, we propose to leverage automatic search techniques to discover new activation functions. Using a combination of exhaustive and reinforcement learning-based search, we discover multiple novel activation functions. We verify the effectiveness of the searches by conducting an empirical evaluation with the best discovered activation function. Our experiments show that the best discovered activation function, f(x) = x · sigmoid(β x), which we name Swish, tends to work better than ReLU on deeper models across a number of challenging datasets. For example, simply replacing ReLUs with Swish units improves top-1 classification accuracy on ImageNet by 0.9% for Mobile NASNet-A and 0.6% for Inception-ResNet-v2. The simplicity of Swish and its similarity to ReLU make it easy for practitioners to replace ReLUs with Swish units in any neural network.

Key facts. Cited by 158 papers in the map (data through 7 Oct 2026). First posted on arXiv 16 Oct 2017.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2018: 1; 2019: 1; 2020: 0; 2021: 30; 2022: 27; 2023: 26; 2024: 35; 2025: 29; 2026: 9.

158 papers.

Sort:

By area:

2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
2024Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Dualitycited by 1,283
2021Alias-Free Generative Adversarial Networkscited by 1,181
2025Cosmos World Foundation Model Platform for Physical AIcited by 833
2021A Survey of Quantization Methods for Efficient Neural Network Inferencecited by 798
2023Retentive Network: A Successor to Transformer for Large Language Modelscited by 594
2023Gated Linear Attention Transformers with Hardware-Efficient Trainingcited by 490
2022Speech Enhancement and Dereverberation with Diffusion-based Generative Modelscited by 195
2022Mega: Moving Average Equipped Gated Attentioncited by 178
2022Self-Stabilization: The Implicit Bias of Gradient Descent at the Edge of Stabilitycited by 151
2022Semantic Image Synthesis via Diffusion Modelscited by 131
2022Understanding Gradient Descent on Edge of Stability in Deep Learningcited by 126

38 more

2023Avatars Grow Legs: Generating Smooth Human Motion from Sparse Tracking Inputs with Diffusion Modelcited by 104
2021A Conditional Point Diffusion-Refinement Paradigm for 3D Point Cloud Completioncited by 100
2021Efficient and Accurate Gradients for Neural SDEscited by 68
2023A Hitchhiker's Guide to Geometric GNNs for 3D Atomic Systemscited by 65
2021Lite Vision Transformer with Enhanced Self-Attentioncited by 64
2021ForceNet: A Graph Neural Network for Large-Scale Quantum Calculationscited by 58
2023RoSteALS: Robust Steganography using Autoencoder Latent Spacecited by 54
2021Rapid training of deep neural networks without skip connections or normalization layers using Deep Kernel Shapingcited by 51
2024A Survey on Visual Mambacited by 50
2024Megalodon: Efficient LLM Pretraining and Inference with Unlimited Context Lengthcited by 46
2023Generative Semantic Communication: Diffusion Models Beyond Bit Recoverycited by 41
2018One-Shot Hierarchical Imitation Learning of Compound Visuomotor Taskscited by 32
2022Revisiting RCAN: Improved Training for Image Super-Resolutioncited by 32
2022Constrained Monotonic Neural Networkscited by 31
2025PENCIL: Long Thoughts with Short Memorycited by 31
2022Real-time Object Detection for Streaming Perceptioncited by 29
2023Why (and When) does Local SGD Generalize Better than SGD?cited by 26
2023The Treasure Beneath Multiple Annotations: An Uncertainty-aware Edge Detectorcited by 25
2021Keep the Gradients Flowing: Using Gradient Flow to Study Sparse Network Optimizationcited by 22
2022Ham2Pose: Animating Sign Language Notation into Pose Sequencescited by 21
2024Semi-Mamba-UNet: Pixel-Level Contrastive and Pixel-Level Cross-Supervised Visual Mamba-based UNet for Semi-Supervised Medical Image Segmentationcited by 20
2021Bolt: Bridging the Gap between Auto-tuners and Hardware-native Performancecited by 18
2024CREMA: Generalizable and Efficient Video-Language Reasoning via Multimodal Modular Fusioncited by 17
2023A Stable and Scalable Method for Solving Initial Value PDEs with Neural Networkscited by 15
2022Centralized Feature Pyramid for Object Detectioncited by 13
2025Do We Always Need the Simplicity Bias? Looking for Optimal Inductive Biases in the Wildcited by 13
2019Zero-shot Imitation Learning from Demonstrations for Legged Robot Visual Navigationcited by 12
2022Topology and geometry of data manifold in deep learningcited by 12
2021Growing Cosine Unit: A Novel Oscillatory Activation Function That Can Speedup Training and Reduce Parameters in Convolutional Neural Networkscited by 11
2022A Survey of Robust 3D Object Detection Methods in Point Cloudscited by 10
20220/1 Deep Neural Networks via Block Coordinate Descentcited by 10
2021Locally-symplectic neural networks for learning volume-preserving dynamicscited by 9
2024Neural Context Flows for Meta-Learning of Dynamical Systemscited by 9
2025Advanced Sign Language Video Generation with Compressed and Quantized Multi-Condition Tokenizationcited by 9
2021Robust Interactive Semantic Segmentation of Pathology Images with Minimal User Inputcited by 8
2021Modeling Design and Control Problems Involving Neural Network Surrogatescited by 8
2021Multimeasurement Generative Modelscited by 8
2023HM-Conformer: A Conformer-based audio deepfake detection system with hierarchical pooling and multi-level classification token aggregation methodscited by 8

50 most cited of 158 All 158 (JSON)