mlsanityData to 14 Oct 2026

Expanded Gating Ranges Improve Activation Functions

arXiv 2405.20768 ↗First posted 25 May 2024cs.NE cs.LGcited by 4

Not on the map. Why?

Abstract (arXiv)

Activation functions are core components of all deep learning architectures. Currently, the most popular activation functions are smooth ReLU variants like GELU and SiLU. These are self-gated activation functions where the range of the gating function is between zero and one. In this paper, we explore the viability of using arctan as a gating mechanism. A self-gated activation function that uses arctan as its gating function has a monotonically increasing first derivative. To make this activation function competitive, it is necessary to introduce a trainable parameter for every MLP block to expand the range of the gating function beyond zero and one. We find that this technique also improves existing self-gated activation functions. We conduct an empirical evaluation of Expanded ArcTan Linear Unit (xATLU), Expanded GELU (xGELU), and Expanded SiLU (xSiLU) and show that they outperform existing activation functions within a transformer architecture. Additionally, expanded gating ranges show promising results in improving first-order Gated Linear Units (GLU).

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 8 of its 20 references that are in the map. First posted on arXiv 25 May 2024.

Builds on

8 of 20 references.

References · 8

  1. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,611
  2. 2016Gaussian Error Linear Units (GELUs)cited by 3,502
  3. 2020GLU Variants Improve Transformercited by 1,804
  4. 2015Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)cited by 1,070
  5. 2017Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learningcited by 950
  6. 2016Language Modeling with Gated Convolutional Networkscited by 702
  7. 2020A survey on modern trainable activation functionscited by 89
  8. 2024ProSparse: Introducing and Enhancing Intrinsic Activation Sparsity within Large Language Modelscited by 45

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 4.

4 papers.

  1. 2026Identifying and Transferring Reasoning-Critical Neurons: Improving LLM Inference Reliability via Activation Steeringcited by 4
  2. 2026IGLU: The Integrated Gaussian Linear Unit Activation Functioncited by 0
  3. 2026Confidence-Adaptive SwiGLU for Mixture-of-Expertscited by 0
  4. 2026Is SwiGLU's Open Positive Tail Necessary? Evidence from Closed-Tail Gating with MemGLUcited by 0