mlsanityData to 7 Oct 2026

Learning Activation Functions to Improve Deep Neural Networks

arXiv 1412.6830 ↗First posted 21 Dec 2014cs.NE cs.CV cs.LGcited by 62

Not on the map. Why?

Abstract (arXiv)

Artificial neural networks typically have a fixed, non-linear activation function at each neuron. We have designed a novel form of piecewise linear activation function that is learned independently for each neuron using gradient descent. With this adaptive activation function, we are able to improve upon deep neural network architectures composed of static rectified linear units, achieving state-of-the-art performance on CIFAR-10 (7.51%), CIFAR-100 (30.83%), and a benchmark from high-energy physics involving Higgs boson decay modes.

Key facts. Cited by 62 papers in the map (data through 7 Oct 2026). First posted on arXiv 21 Dec 2014.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 11; 2022: 12; 2023: 12; 2024: 13; 2025: 4; 2026: 10.

62 papers.

Sort:

  1. 2021Activation Functions in Deep Learning: A Comprehensive Survey and Benchmarkcited by 153
  2. 2022Evaluating the Susceptibility of Pre-Trained Language Models via Handcrafted Adversarial Examplescited by 65
  3. 2021Deep Kronecker neural networks: A general framework for neural networks with adaptive activation functionscited by 49
  4. 2023When Deep Learning Meets Polyhedral Theory: A Surveycited by 39
  5. 2022How important are activation functions in regression and classification? A survey, performance comparison, and future directionscited by 29
  6. 2022Improving Lipschitz-Constrained Neural Networks by Learning Activation Functionscited by 23
  7. 2022On the Number of Regions of Piecewise Linear Neural Networkscited by 21
  8. 2023Learning Specialized Activation Functions for Physics-informed Neural Networkscited by 14
  9. 2021Privacy and Trust Redefined in Federated Machine Learningcited by 13
  10. 2025Quantum Variational Activation Functions Empower Kolmogorov-Arnold Networkscited by 10
  11. 2021Learning specialized activation functions with the Piecewise Linear Unitcited by 7
  12. 2022Piecewise Linear Neural Networks and Deep Learningcited by 7
38 more
  1. 2023Efficient Activation Function Optimization through Surrogate Modelingcited by 5
  2. 2024Kolmogorov-Arnold Networks in Low-Data Regimes: A Comparative Study with Multilayer Perceptronscited by 5
  3. 2022Optimal Activation Functions for the Random Features Regression Modelcited by 4
  4. 2022Neural Networks with A La Carte Selection of Activation Functionscited by 4
  5. 2023Increasing biases can be more efficient than increasing weightscited by 4
  6. 2023ENN: A Neural Network with DCT Adaptive Activation Functionscited by 4
  7. 2023Homotopy Relaxation Training Algorithms for Infinite-Width Two-Layer ReLU Neural Networkscited by 4
  8. 2024KAN we improve on HEP classification tasks? Kolmogorov-Arnold Networks applied to an LHC physics examplecited by 4
  9. 2025Don't Forget the Nonlinearity: Unlocking Activation Functions in Efficient Fine-Tuningcited by 4
  10. 2022Bayesian Neural Networks for Macroeconomic Analysiscited by 3
  11. 2023Optimal Nonlinearities Improve Generalization Performance of Random Featurescited by 3
  12. 2024Adaptive Activation Functions for Predictive Modeling with Sparse Experimental Datacited by 3
  13. 2026More Expressive Feedforward Layers: Part I. Token-Adaptive Mixing of Activationscited by 3
  14. 2021Faster Convergence in Deep-Predictive-Coding Networks to Learn Deeper Representationscited by 2
  15. 2021Comparison of different convolutional neural network activation functions and methods for building ensemblescited by 2
  16. 2022Neural Network Verification as Piecewise Linear Optimization: Formulations for the Composition of Staircase Functionscited by 2
  17. 2023Empirical study of the modulus as activation function in computer vision applicationscited by 2
  18. 2024From KAN to GR-KAN: Advancing Speech Enhancement with KAN-Based Methodologycited by 2
  19. 2021Deep ensembles based on Stochastic Activation Selection for Polyp Segmentationcited by 1
  20. 2021Piecewise Linear Units Improve Deep Neural Networkscited by 1
  21. 2024Physics-informed Deep Learning to Solve Three-dimensional Terzaghi Consolidation Equation: Forward and Inverse Problemscited by 1
  22. 2024An In-Depth Analysis of Data Reduction Methods for Sustainable Deep Learningcited by 1
  23. 2024Interpolating neural network: A novel unification of machine learning and interpolation theorycited by 1
  24. 2024Efficient Search for Customized Activation Functions with Gradient Descentcited by 1
  25. 2024Controlled Learning of Pointwise Nonlinearities in Neural-Network-Like Architecturescited by 1
  26. 2021Tensor-based framework for training flexible neural networkscited by 0
  27. 2021PowerLinear Activation Functions with application to the first layer of CNNscited by 0
  28. 2021Using Computational Intelligence for solving the Ornstein-Zernike equationcited by 0
  29. 2022Neuronal diversity can improve machine learning for physics and beyondcited by 0
  30. 2022Deep Learning Models of the Discrete Component of the Galactic Interstellar Gamma-Ray Emissioncited by 0
  31. 2022Tackling Multiple Tasks with One Single Learning Frameworkcited by 0
  32. 2023Optimizing Neural Networks through Activation Function Discovery and Automatic Weight Initializationcited by 0
  33. 2023A Melting Pot of Evolution and Learningcited by 0
  34. 2023Optimizing Performance of Feedforward and Convolutional Neural Networks through Dynamic Activation Functionscited by 0
  35. 2023Linear Oscillation: A Novel Activation Function for Vision Transformercited by 0
  36. 2024A Novel Paradigm for Neural Computation: X-Net with Learnable Neurons and Adaptable Structurecited by 0
  37. 2024Contrast-Enhanced Gating in GRUs for Robust Low-Data Sequence Learningcited by 0
  38. 2024Nonlinearity Enhanced Adaptive Activation Functionscited by 0

50 most cited of 62 All 62 (JSON)