mlsanityData to 7 Oct 2026

Pad'e Activation Units: End-to-end Learning of Flexible Activation Functions in Deep Networks

arXiv 1907.06732 ↗First posted 15 Jul 2019cs.LG cs.NEcited by 65

Not on the map. Why?

Abstract (arXiv)

The performance of deep network learning strongly depends on the choice of the non-linear activation function associated with each neuron. However, deciding on the best activation is non-trivial, and the choice depends on the architecture, hyper-parameters, and even on the dataset. Typically these activations are fixed by hand before training. Here, we demonstrate how to eliminate the reliance on first picking fixed activation functions by using flexible parametric rational functions instead. The resulting Pad'e Activation Units (PAUs) can both approximate common activation functions and also learn new ones while providing compact representations. Our empirical evidence shows that end-to-end learning deep networks with PAUs can increase the predictive performance. Moreover, PAUs pave the way to approximations with provable robustness. https://github.com/ml-research/pau

Key facts. Cited by 65 papers in the map (data through 7 Oct 2026). First posted on arXiv 15 Jul 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 11; 2022: 4; 2023: 12; 2024: 12; 2025: 15; 2026: 11.

65 papers.

Sort:

By area:

  1. 2023Convolutional Neural Operators for robust and accurate learning of PDEscited by 207
  2. 2021Activation Functions in Deep Learning: A Comprehensive Survey and Benchmarkcited by 153
  3. 2024Kolmogorov-Arnold Transformercited by 56
  4. 2023Class-Incremental Exemplar Compression for Class-Incremental Learningcited by 50
  5. 2023A Machine Learning-oriented Survey on Tiny Machine Learningcited by 26
  6. 2021Adaptive Rational Activations to Boost Deep Reinforcement Learningcited by 23
  7. 2022Deep Learning and Symbolic Regression for Discovering Parametric Equationscited by 22
  8. 2024MiLoRA: Efficient Mixture of Low-Rank Adaptation for Large Language Models Fine-tuningcited by 19
  9. 2021Sisyphus: A Cautionary Tale of Using Low-Degree Polynomial Activations in Privacy-Preserving Deep Learningcited by 14
  10. 2023Learning Specialized Activation Functions for Physics-informed Neural Networkscited by 14
  11. 2022Adaptable Adapterscited by 13
  12. 2023Self-Expanding Neural Networkscited by 13
38 more
  1. 2025Quantum Variational Activation Functions Empower Kolmogorov-Arnold Networkscited by 10
  2. 2021OSS-Net: Memory Efficient High Resolution Semantic Segmentation of 3D Medical Datacited by 9
  3. 2024PEDRO: Parameter-Efficient Fine-tuning with Prompt DEpenDent Representation MOdificationcited by 8
  4. 2021Learning specialized activation functions with the Piecewise Linear Unitcited by 7
  5. 2021SMU: smooth activation function for deep networks using smoothing maximum techniquecited by 7
  6. 2023Covering Number of Real Algebraic Varieties and Beyond: Improved Bounds and Applicationscited by 7
  7. 2025Activation Function Design Sustains Plasticity in Continual Learningcited by 6
  8. 2021ErfAct and Pserf: Non-monotonic Smooth Trainable Activation Functionscited by 5
  9. 2022Transformers with Learnable Activation Functionscited by 5
  10. 2023Efficient Activation Function Optimization through Surrogate Modelingcited by 5
  11. 2021Orthogonal-Pad'e Activation Functions: Trainable Activation functions for smooth and faster convergence in deep networkscited by 4
  12. 2023A Non-monotonic Smooth Activation Functioncited by 3
  13. 2025KAN or MLP? Point Cloud Shows the Way Forwardcited by 3
  14. 2025FlashKAT: Understanding and Addressing Performance Bottlenecks in the Kolmogorov-Arnold Transformercited by 3
  15. 2025CoFrNets: Interpretable Neural Architecture Inspired by Continued Fractionscited by 3
  16. 2023Improved Auto-Encoding using Deterministic Projected Belief Networkscited by 2
  17. 2024PAON: A New Neuron Model using Pad'e Approximantscited by 2
  18. 2024Revisiting the Equivalence of Bayesian Neural Networks and Gaussian Processes: On the Importance of Learning Activationscited by 2
  19. 2024Graph Neural Networks Need Cluster-Normalize-Activate Modulescited by 2
  20. 2024From KAN to GR-KAN: Advancing Speech Enhancement with KAN-Based Methodologycited by 2
  21. 2025Balancing Expressivity and Robustness: Constrained Rational Activations for Reinforcement Learningcited by 2
  22. 2026Rational ANOVA Networkscited by 2
  23. 2021Legendre Deep Neural Network (LDNN) and its application for approximation of nonlinear Volterra Fredholm Hammerstein integral equationscited by 1
  24. 2021Effect of the output activation function on the probabilities and errors in medical image segmentationcited by 1
  25. 2021SAU: Smooth activation function using convolution with approximate identitiescited by 1
  26. 2024IAPT: Instruction-Aware Prompt Tuning for Large Language Modelscited by 1
  27. 2024Hadamard Representation: Scaffolding Performance Across Model-free RLcited by 1
  28. 2024Efficient Search for Customized Activation Functions with Gradient Descentcited by 1
  29. 2024Taming the Tail: Leveraging Asymmetric Loss and Pade Approximation to Overcome Medical Image Long-Tailed Class Imbalancecited by 1
  30. 2025NLP Methods May Actually Be Better Than Professors at Estimating Question Difficultycited by 1
  31. 2025PO-CKAN:Physics Informed Deep Operator Kolmogorov Arnold Networks with Chunk Rational Structurecited by 1
  32. 2026Rational Neural Networks have Expressivity Advantagescited by 1
  33. 2026A Universal Reproducing Kernel Hilbert Space from Polynomial Alignment and IMQ Distancecited by 1
  34. 2022Deepening Neural Networks Implicitly and Locally via Recurrent Attention Strategycited by 0
  35. 2023A comparison of rational and neural network based approximationscited by 0
  36. 2023Optimizing Neural Networks through Activation Function Discovery and Automatic Weight Initializationcited by 0
  37. 2023Rational Neural Network Controllerscited by 0
  38. 2024Activations Through Extensions: A Framework To Boost Performance Of Neural Networkscited by 0

50 most cited of 65 All 65 (JSON)