mlsanityData to 7 Oct 2026

Trainable Highly-expressive Activation Functions

arXiv 2407.07564 ↗First posted 10 Jul 2024cs.CVcited by 6

Not on the map. Why?

Abstract (arXiv)

Nonlinear activation functions are pivotal to the success of deep neural nets, and choosing the appropriate activation function can significantly affect their performance. Most networks use fixed activation functions (e.g., ReLU, GELU, etc.), and this choice might limit their expressiveness. Furthermore, different layers may benefit from diverse activation functions. Consequently, there has been a growing interest in trainable activation functions. In this paper, we introduce DiTAC, a trainable highly-expressive activation function based on an efficient diffeomorphic transformation (called CPAB). Despite introducing only a negligible number of trainable parameters, DiTAC enhances model expressiveness and performance, often yielding substantial improvements. It also outperforms existing activation functions (regardless whether the latter are fixed or trainable) in tasks such as semantic segmentation, image generation, regression problems, and image classification. Our code is available at https://github.com/BGU-CS-VIL/DiTAC.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 40 of its 68 references that are in the map. First posted on arXiv 10 Jul 2024.

Builds on

40 of 68 references.

References · 40

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  3. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  4. 2016The Cityscapes Dataset for Semantic Urban Scene Understandingcited by 4,427
  5. 2014Deep Learning Face Attributes in the Wildcited by 3,679
  6. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,590
  7. 2022A ConvNet for the 2020scited by 3,545
  8. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  9. 2016DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFscited by 3,239
  10. 2016Improved Techniques for Training GANscited by 2,938
  11. 2018Large Scale GAN Training for High Fidelity Natural Image Synthesiscited by 2,609
  12. 2016Pyramid Scene Parsing Networkcited by 2,472
28 more
  1. 2015Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networkscited by 2,048
  2. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,031
  3. 2019Searching for MobileNetV3cited by 1,984
  4. 2015Spatial Transformer Networkscited by 1,423
  5. 2018Unified Perceptual Parsing for Scene Understandingcited by 1,265
  6. 2016Semantic Understanding of Scenes through the ADE20K Datasetcited by 1,063
  7. 2015Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)cited by 1,061
  8. 2017Self-Normalizing Neural Networkscited by 591
  9. 2021A White Paper on Neural Network Quantizationcited by 453
  10. 2021ResNet strikes back: An improved training procedure in timmcited by 365
  11. 2013Maxout Networkscited by 186
  12. 2021Activation Functions in Deep Learning: A Comprehensive Survey and Benchmarkcited by 153
  13. 2020A survey on modern trainable activation functionscited by 89
  14. 2017Continuously Differentiable Exponential Linear Unitscited by 73
  15. 2020Dynamic ReLUcited by 55
  16. 2021Deep Kronecker neural networks: A general framework for neural networks with adaptive activation functionscited by 49
  17. 2016Parametric Exponential Linear Unit for Deep Convolutional Neural Networkscited by 32
  18. 2015Dreaming More Data: Class-dependent Distributions over Diffeomorphisms for Learned Data Augmentationcited by 24
  19. 2020Activate or Not: Learning Customized Activationcited by 21
  20. 2019Explicit Disentanglement of Appearance and Perspective in Generative Modelscited by 20
  21. 2022Wavelet Feature Maps Compression for Image-to-Image CNNscited by 16
  22. 2024Continuous Piecewise-Affine Based Motion Model for Image Animationcited by 8
  23. 2020Probabilistic Spatial Transformer Networkscited by 7
  24. 2022Closed-Form Diffeomorphic Transformations for Time Series Alignmentcited by 7
  25. 2013Improving Deep Neural Networks with Probabilistic Maxout Unitscited by 4
  26. 2022Stochastic Adaptive Activation Functioncited by 4
  27. 2022Leveraging Statistical Shape Priors in GAN-based ECG Synthesiscited by 4
  28. 2023ErfReLU: Adaptive Activation Function for Deep Neural Networkcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 5; 2026: 1.

6 papers.

  1. 2025Do We Always Need the Simplicity Bias? Looking for Optimal Inductive Biases in the Wildcited by 13
  2. 2025DiTASK: Multi-Task Fine-Tuning with Diffeomorphic Transformationscited by 7
  3. 2025Diffeomorphic Temporal Alignment Nets for Time-series Joint Alignment and Averagingcited by 3
  4. 2025TimePoint: Accelerated Time Series Alignment via Self-Supervised Keypoint and Descriptor Learningcited by 2
  5. 2026Can Transformers Really Do It All? On the Compatibility of Inductive Biases Across Taskscited by 2
  6. 2025Developing Training Procedures for Piecewise-linear Spline Activation Functions in Neural Networkscited by 0