# Activate or Not: Learning Customized Activation

arXiv: https://arxiv.org/abs/2009.04759 · HTML: https://mlsanity.com/paper/2009.04759/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 Sep 2020.

## Built on by (most cited first, 22 of 22)

- [Layer Folding: Neural Network Depth Reduction using Activation Linearization](https://mlsanity.com/paper/2106.09309/) (2021)
- [Learning Specialized Activation Functions for Physics-informed Neural Networks](https://mlsanity.com/paper/2308.04073/) (2023)
- [Trainable Highly-expressive Activation Functions](https://mlsanity.com/paper/2407.07564/) (2024)
- [TE-YOLOF: Tiny and efficient YOLOF for blood cell detection](https://mlsanity.com/paper/2108.12313/) (2021)
- [ErfAct and Pserf: Non-monotonic Smooth Trainable Activation Functions](https://mlsanity.com/paper/2109.04386/) (2021)
- [Human-Inspired Facial Sketch Synthesis with Dynamic Adaptation](https://mlsanity.com/paper/2309.00216/) (2023)
- [Exploring Linear Feature Disentanglement For Neural Networks](https://mlsanity.com/paper/2203.11700/) (2022)
- [MorphoActivation: Generalizing ReLU activation function by mathematical morphology](https://mlsanity.com/paper/2207.06413/) (2022)
- [Effect of the output activation function on the probabilities and errors in medical image segmentation](https://mlsanity.com/paper/2109.00903/) (2021)
- [A Medical Pre-Diagnosis System for Histopathological Image of Breast Cancer](https://mlsanity.com/paper/2109.07878/) (2021)
- [SAU: Smooth activation function using convolution with approximate identities](https://mlsanity.com/paper/2109.13210/) (2021)
- [Learning Activation Functions for Sparse Neural Networks](https://mlsanity.com/paper/2305.10964/) (2023)
- [Dynamic Shuffle: An Efficient Channel Mixture Method](https://mlsanity.com/paper/2310.02776/) (2023)
- [DrivAer Transformer: A high-precision and fast prediction method for vehicle aerodynamic drag coefficient based on the DrivAerNet++ dataset](https://mlsanity.com/paper/2504.08217/) (2025)
- [First Power Linear Unit with Sign](https://mlsanity.com/paper/2111.14349/) (2021)
- [RepAct: The Re-parameterizable Adaptive Activation Function](https://mlsanity.com/paper/2407.00131/) (2024)
- [Swish-T : Enhancing Swish Activation with Tanh Bias for Improved Neural Network Performance](https://mlsanity.com/paper/2407.01012/) (2024)
- [Activations Through Extensions: A Framework To Boost Performance Of Neural Networks](https://mlsanity.com/paper/2408.03599/) (2024)
- [An Appearance Defect Detection Method for Cigarettes Based on C-CenterNet](https://mlsanity.com/paper/2502.06119/) (2025)
- [AcTTA: Rethinking Test-Time Adaptation via Dynamic Activation](https://mlsanity.com/paper/2603.26096/) (2026)
- [AutoNorm: Understanding Adaptive Normalization in Transformers through Differentiable Gating](https://mlsanity.com/paper/2607.10593/) (2026)
- [Rethinking Activation Functions: Reducing Bias Can Improve Learning](https://mlsanity.com/paper/2610.14500/) (2026)

## Abstract (arXiv)

We present a simple, effective, and general activation function we term ACON which learns to activate the neurons or not. Interestingly, we find Swish, the recent popular NAS-searched activation, can be interpreted as a smooth approximation to ReLU. Intuitively, in the same way, we approximate the more general Maxout family to our novel ACON family, which remarkably improves the performance and makes Swish a special case of ACON. Next, we present meta-ACON, which explicitly learns to optimize the parameter switching between non-linear (activate) and linear (inactivate) and provides a new design space. By simply changing the activation function, we show its effectiveness on both small models and highly optimized large models (e.g. it improves the ImageNet top-1 accuracy rate by 6.7% and 1.8% on MobileNet-0.25 and ResNet-152, respectively). Moreover, our novel ACON can be naturally transferred to object detection and semantic segmentation, showing that ACON is an effective alternative in a variety of tasks. Code is available at https://github.com/nmaac/acon.