mlsanityData to 14 Oct 2026

Periodic Activation Functions Induce Stationarity

arXiv 2110.13572 ↗First posted 26 Oct 2021cs.LG stat.MLcited by 21

Not on the map. Why?

Abstract (arXiv)

Neural network models are known to reinforce hidden data biases, making them unreliable and difficult to interpret. We seek to build models that `know what they do not know' by introducing inductive biases in the function space. We show that periodic activation functions in Bayesian neural networks establish a connection between the prior on the network weights and translation-invariant, stationary Gaussian process priors. Furthermore, we show that this link goes beyond sinusoidal (Fourier) activations by also covering triangular wave and periodic ReLU activation functions. In a series of experiments, we show that periodic activation functions obtain comparable performance for in-domain data and capture sensitivity to perturbed inputs in deep neural networks for out-of-domain detection.

Key facts. Cited by 21 papers in the map (data through 14 Oct 2026). Builds on 28 of its 63 references that are in the map. First posted on arXiv 26 Oct 2021.

Builds on

28 of 63 references.

References · 28

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  2. 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,448
  3. 2014Going Deeper with Convolutionscited by 4,241
  4. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,611
  5. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
  6. 2020Implicit Neural Representations with Periodic Activation Functionscited by 2,238
  7. 2020Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domainscited by 2,200
  8. 2014Deep Learning in Neural Networks: An Overviewcited by 770
  9. 2017Deep Neural Networks as Gaussian Processescited by 622
  10. 2019A Simple Baseline for Bayesian Uncertainty in Deep Learningcited by 560
  11. 2021FNet: Mixing Tokens with Fourier Transformscited by 347
  12. 2018Why ReLU networks yield high-confidence predictions far away from the training data and how to mitigate the problemcited by 323
16 more
  1. 2018Gaussian Process Behaviour in Wide Deep Neural Networkscited by 317
  2. 2020Being Bayesian, Even Just a Bit, Fixes Overconfidence in ReLU Networkscited by 231
  3. 2020The Next Decade in AI: Four Steps Towards Robust Artificial Intelligencecited by 195
  4. 2016GPflow: A Gaussian process library using TensorFlowcited by 140
  5. 2019Functional Variational Bayesian Neural Networkscited by 127
  6. 2021Bayesian Neural Network Priors Revisitedcited by 121
  7. 2014Hilbert Space Methods for Reduced-Rank Gaussian Process Regressioncited by 88
  8. 2016Variational Fourier features for Gaussian processescited by 85
  9. 2015On the Error of Random Fourier Featurescited by 76
  10. 2020A statistical theory of cold posteriors in deep neural networkscited by 54
  11. 2020Know Your Limits: Uncertainty Estimation with ReLU Classifiers Fails at Reliable OOD Detectioncited by 26
  12. 2019Expressive Priors in Bayesian Neural Networks: Kernel Combinations and Periodic Functionscited by 22
  13. 2019Fourier Neural Networks: A Comparative Studycited by 14
  14. 2020Predictive Complexity Priorscited by 14
  15. 2020Stationary Activations for Uncertainty Calibration in Deep Learningcited by 14
  16. 2017Invariance of Weight Distributions in Rectified MLPscited by 12

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 3; 2023: 6; 2024: 4; 2025: 5; 2026: 3.

21 papers.

Sort:

By area:

  1. 2022VeLO: Training Versatile Learned Optimizers by Scaling Upcited by 57
  2. 2023Coherent Soft Imitation Learningcited by 22
  3. 2022Intrinsic Neural Fields: Learning Functions on Manifoldscited by 20
  4. 2023Fixing Overconfidence in Dynamic Neural Networkscited by 19
  5. 2023Squared Neural Families: A New Class of Tractable Density Modelscited by 12
  6. 2023Beyond Unimodal: Generalising Neural Processes for Multimodal Uncertainty Estimationcited by 11
  7. 2024Fast Training of Sinusoidal Neural Fields via Scaling Initializationcited by 11
  8. 2023Promises and Pitfalls of the Linearized Laplace in Bayesian Optimizationcited by 10
  9. 2023Collapsed Inference for Bayesian Deep Learningcited by 6
  10. 2024Streamlining Prediction in Bayesian Deep Learningcited by 5
  11. 2022Deep equilibrium models as estimators for continuous latent variablescited by 4
  12. 2024Physics-Informed Neural Networks: Minimizing Residual Loss with Wide Networks and Effective Activationscited by 4
9 more
  1. 2025Quantum advantage for learning shallow neural networks with natural data distributionscited by 3
  2. 2024Revisiting the Equivalence of Bayesian Neural Networks and Gaussian Processes: On the Importance of Learning Activationscited by 2
  3. 2025Barron Space Representations for Elliptic PDEs with Homogeneous Boundary Conditionscited by 1
  4. 2026XQCfD: Accelerating Fast Actor-Critic Algorithms with Prior Data and Prior Policiescited by 1
  5. 2025DPN-GAN: Inducing Periodic Activations in Generative Adversarial Networks for High-Fidelity Audio Synthesiscited by 0
  6. 2025Exploration Behavior of Untrained Policiescited by 0
  7. 2025Hi-fi functional priors by learning activationscited by 0
  8. 2026HOSC: A Periodic Activation with Saturation Control for High-Fidelity Implicit Neural Representationscited by 0
  9. 2026Coherent Off-Policy Improvement of Large Behavior Models with Learned Rewardscited by 0