mlsanityData to 7 Oct 2026

Reproducing Activation Function for Deep Learning

arXiv 2101.04844 ↗First posted 13 Jan 2021cs.LG cs.CV cs.NAcited by 19

Not on the map. Why?

Abstract (arXiv)

We propose reproducing activation functions (RAFs) to improve deep learning accuracy for various applications ranging from computer vision to scientific computing. The idea is to employ several basic functions and their learnable linear combination to construct neuron-wise data-driven activation functions for each neuron. Armed with RAFs, neural networks (NNs) can reproduce traditional approximation tools and, therefore, approximate target functions with a smaller number of parameters than traditional NNs. In NN training, RAFs can generate neural tangent kernels (NTKs) with a better condition number than traditional activation functions lessening the spectral bias of deep learning. As demonstrated by extensive numerical tests, the proposed RAFs can facilitate the convergence of deep learning optimization for a solution with higher accuracy than existing deep learning solvers for audio/image/video reconstruction, PDEs, and eigenvalue problems. With RAFs, the errors of audio/video reconstruction, PDEs, and eigenvalue problems are decreased by over 14%, 73%, 99%, respectively, compared with baseline, while the performance of image reconstruction increases by 58%.

Key facts. Cited by 19 papers in the map (data through 7 Oct 2026). Builds on 51 of its 83 references that are in the map. First posted on arXiv 13 Jan 2021.

Builds on

51 of 83 references.

References · 51

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2019DeepSDF: Learning Continuous Signed Distance Functions for Shape Representationcited by 2,682
  4. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  5. 2020Implicit Neural Representations with Periodic Activation Functionscited by 2,214
  6. 2020Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domainscited by 2,170
  7. 2018Learning Implicit Fields for Generative Shape Modelingcited by 1,060
  8. 2019Scene Representation Networks: Continuous 3D-Structure-Aware Neural Scene Representationscited by 864
  9. 2019On the Convergence of Adam and Beyondcited by 834
  10. 2019PIFu: Pixel-Aligned Implicit Function for High-Resolution Clothed Human Digitizationcited by 818
  11. 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 784
  12. 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 732
38 more
  1. 2017DGM: A deep learning algorithm for solving partial differential equationscited by 674
  2. 2018Gradient Descent Provably Optimizes Over-parameterized Neural Networkscited by 655
  3. 2020When and why PINNs fail to train: A neural tangent kernel perspectivecited by 629
  4. 2017The Deep Ritz method: A deep learning-based numerical algorithm for solving variational problemscited by 581
  5. 2016Error bounds for approximations with deep ReLU networkscited by 567
  6. 2019Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networkscited by 562
  7. 2017Solving high-dimensional partial differential equations using deep learningcited by 549
  8. 2019Frequency Principle: Fourier Analysis Sheds Light on Deep Neural Networkscited by 360
  9. 2019DIST: Rendering Deep Implicit Signed Distance Function with Differentiable Sphere Tracingcited by 179
  10. 2019Towards Understanding the Spectral Bias of Deep Learningcited by 173
  11. 2018Optimal approximation of continuous functions by very deep ReLU networkscited by 165
  12. 2020Deep Network Approximation for Smooth Functionscited by 163
  13. 2016Why and When Can Deep -- but Not Shallow -- Networks Avoid the Curse of Dimensionality: a Reviewcited by 155
  14. 2019NASA: Neural Articulated Shape Approximationcited by 149
  15. 2017A unified deep artificial neural network approach to partial differential equations in complex geometriescited by 122
  16. 2019Deep Network Approximation Characterized by Number of Neuronscited by 120
  17. 2019The phase diagram of approximation rates for deep neural networkscited by 86
  18. 2019How Much Over-parameterization Is Sufficient to Learn Deep ReLU Networks?cited by 84
  19. 2020Multi-scale Deep Neural Network (MscaleDNN) for Solving Poisson-Boltzmann Equation in Complex Domainscited by 78
  20. 2018A Priori Estimates of the Population Risk for Two-layer Neural Networkscited by 76
  21. 2020Rational neural networkscited by 75
  22. 2020Neural Network Approximation: Three Hidden Layers Are Enoughcited by 63
  23. 2019Reconstructing continuous distributions of 3D protein structure from cryo-EM imagescited by 56
  24. 2019Error bounds for deep ReLU networks using the Kolmogorov--Arnold superposition theoremcited by 49
  25. 2019Theory of the Frequency Principle for General Deep Neural Networkscited by 45
  26. 2018Analysis of the Generalization Error: Empirical Risk Minimization over Deep Artificial Neural Networks Overcomes the Curse of Dimensionality in the Numerical Approximation of Black-Scholes Partial Differential Equationscited by 44
  27. 2018Convergence of the Deep BSDE Method for Coupled FBSDEscited by 42
  28. 2020Two-Layer Neural Networks for Partial Differential Equations: Optimization and Generalization Theorycited by 41
  29. 2019Adaptive Deep Learning for High-Dimensional Hamilton-Jacobi-Bellman Equationscited by 40
  30. 2017Geometry of Optimization and Implicit Regularization in Deep Learningcited by 39
  31. 2018Exponential Convergence of the Deep Neural Network Approximation for Analytic Functionscited by 38
  32. 2020Solving high-dimensional eigenvalue problems using deep neural networks: A diffusion Monte Carlo like approachcited by 35
  33. 2019A Phase Shift Deep Neural Network for High Frequency Approximation and Wave Problemscited by 34
  34. 2020On the convergence of physics informed neural networks for linear second-order elliptic and parabolic type PDEscited by 31
  35. 2019Deep least-squares methods: an unsupervised learning-based numerical method for solving elliptic PDEscited by 27
  36. 2019A Priori Estimates of the Population Risk for Residual Networkscited by 14
  37. 2018Towards Theoretical Understanding of Large Batch Training in Stochastic Gradient Descentcited by 6
  38. 2018Implicit Regularization of Stochastic Gradient Descent in Natural Language Processing: Observations and Implicationscited by 4

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 2; 2022: 4; 2023: 3; 2024: 6; 2025: 3; 2026: 1.

19 papers.

Sort:

By area:

  1. 2022Respecting causality is all you need for training physics-informed neural networkscited by 154
  2. 2021Beyond Periodicity: Towards a Unifying Framework for Activations in Coordinate-MLPscited by 134
  3. 2024Operator Learning: Algorithms and Analysiscited by 84
  4. 2022Overview frequency principle/spectral bias in deep learningcited by 68
  5. 2022Neural Fourier Filter Bankcited by 29
  6. 2025Mitigating Spectral Bias in Neural Operators via High-Frequency Scaling for Physical Systemscited by 29
  7. 2024MeanSparse: Post-Training Robustness Enhancement Through Mean-Centered Feature Sparsificationcited by 20
  8. 2022Learning a Single Neuron for Non-monotonic Activation Functionscited by 13
  9. 2023On the Optimal Expressive Power of ReLU DNNs and Its Application in Approximation with Kolmogorov Superposition Theoremcited by 13
  10. 2021Solving PDEs on Unknown Manifolds with Machine Learningcited by 12
  11. 2023Deep Neural Networks and Finite Elements of Any Order on Arbitrary Dimensionscited by 8
  12. 2024Learnable Activation Functions in Physics-Informed Neural Networks for Solving Partial Differential Equationscited by 7
7 more
  1. 2023A Sequential Meta-Transfer (SMT) Learning to Combat Complexities of Physics-Informed Neural Networks: Application to Composites Autoclave Processingcited by 3
  2. 2024Towards the Spectral bias Alleviation by Normalizations in Coordinate Networkscited by 2
  3. 2024Learning Epidemiological Dynamics via the Finite Expression Methodcited by 2
  4. 2024Activations Through Extensions: A Framework To Boost Performance Of Neural Networkscited by 0
  5. 2025Frequency-adaptive tensor neural networks for high-dimensional multi-scale problemscited by 0
  6. 2025Multi-Scale Finite Expression Method for PDEs with Oscillatory Solutions on Complex Domainscited by 0
  7. 2026Improving Variational Autoencoder using Random Fourier Transformation: An Aviation Safety Anomaly Detection Case-Studycited by 0