mlsanityData to 14 Oct 2026

Deriving Activation Functions Using Integration

arXiv 2411.13010 ↗First posted 20 Nov 2024cs.LG cs.NEcited by 5

Not on the map. Why?

Abstract (arXiv)

Our work proposes a novel approach to designing activation functions by focusing on their gradients and deriving the corresponding activation functions using integration. We introduce the Expanded Integral of the Exponential Linear Unit (xIELU), a trainable piecewise activation function derived by integrating trainable affine transformations applied to the Exponential Linear Unit (ELU). xIELU combines two key properties for the gradient: (1) a trainable and linearly increasing gradient for positive inputs, similar to Squared ReLU (ReLU2), and (2) a trainable gradient that can take negative values for negative inputs, inspired by Expanded SiLU (xSiLU). Conceptually, xIELU can be viewed as an extension of ReLU2 to handle negative inputs. The trainable parameters in xIELU allow it to adaptively reduce its nonlinearity for higher-level representations deeper in the network. In experiments with 1.1B and 3B parameter Llama models trained on 125B tokens of FineWeb Edu, xIELU achieves lower perplexity compared to popular activation functions like ReLU2 and SwiGLU when matched for the same compute cost and parameter count. A reference implementation is available at https://github.com/Anonymous5823/xielu.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 14 of its 26 references that are in the map. First posted on arXiv 20 Nov 2024.

Builds on

14 of 26 references.

References · 14

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2014Going Deeper with Convolutionscited by 4,241
  4. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,611
  5. 2016Gaussian Error Linear Units (GELUs)cited by 3,502
  6. 2020GLU Variants Improve Transformercited by 1,804
  7. 2019Root Mean Square Layer Normalizationcited by 1,480
  8. 2015Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)cited by 1,070
  9. 2017Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learningcited by 950
  10. 2024MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategiescited by 723
  11. 2024DeepSeek LLM: Scaling Open-Source Language Models with Longtermismcited by 642
  12. 2017Self-Normalizing Neural Networkscited by 598
2 more
  1. 2021Primer: Searching for Efficient Transformers for Language Modelingcited by 164
  2. 2024Scaling FP8 training to trillion-token LLMscited by 65

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 3; 2026: 2.

5 papers.

  1. 2025Towards Fully FP8 GEMM LLM Training at Scalecited by 11
  2. 2025Speedrunning ImageNet Diffusioncited by 3
  3. 2026Apertus LLM Family Expansion via Distillation and Quantizationcited by 1
  4. 2025Comparative Analysis of CNN and Transformer Architectures with Heart Cycle Normalization for Automated Phonocardiogram Classificationcited by 0
  5. 2026Is SwiGLU's Open Positive Tail Necessary? Evidence from Closed-Tail Gating with MemGLUcited by 0