mlsanityData to 7 Oct 2026

Frequency and Generalisation of Periodic Activation Functions in Reinforcement Learning

arXiv 2407.06756 ↗First posted 9 Jul 2024cs.LG cs.AI cs.NEcited by 4

Not on the map. Why?

Abstract (arXiv)

Periodic activation functions, often referred to as learned Fourier features have been widely demonstrated to improve sample efficiency and stability in a variety of deep RL algorithms. Potentially incompatible hypotheses have been made about the source of these improvements. One is that periodic activations learn low frequency representations and as a result avoid overfitting to bootstrapped targets. Another is that periodic activations learn high frequency representations that are more expressive, allowing networks to quickly fit complex value functions. We analyse these claims empirically, finding that periodic representations consistently converge to high frequencies regardless of their initialisation frequency. We also find that while periodic activation functions improve sample efficiency, they exhibit worse generalization on states with added observation noise -- especially when compared to otherwise equivalent networks with ReLU activation functions. Finally, we show that weight decay regularization is able to partially offset the overfitting of periodic activation functions, delivering value functions that learn quickly while also generalizing.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 28 of its 49 references that are in the map. First posted on arXiv 9 Jul 2024.

Builds on

28 of 49 references.

References · 28

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  2. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  3. 2020Fourier Neural Operator for Parametric Partial Differential Equationscited by 2,806
  4. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  5. 2020Implicit Neural Representations with Periodic Activation Functionscited by 2,214
  6. 2020Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domainscited by 2,170
  7. 2018On the Spectral Bias of Neural Networkscited by 1,321
  8. 2018DeepMind Control Suitecited by 935
  9. 2023Progress measures for grokking via mechanistic interpretabilitycited by 822
  10. 2016Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networkscited by 541
  11. 2018Spherical CNNscited by 452
  12. 2018Quantifying Generalization in Reinforcement Learningcited by 345
16 more
  1. 2020Agent57: Outperforming the Atari Human Benchmarkcited by 300
  2. 2016Exponential expressivity in deep neural networks through transient chaoscited by 260
  3. 2022The Primacy Bias in Deep Reinforcement Learningcited by 242
  4. 2018Deep Reinforcement Learning and the Deadly Triadcited by 170
  5. 2021A Survey of Zero-shot Generalisation in Deep Reinforcement Learningcited by 164
  6. 2020Implicit Under-Parameterization Inhibits Data-Efficient Deep Reinforcement Learningcited by 134
  7. 2016Deep Successor Reinforcement Learningcited by 72
  8. 2021Rapid training of deep neural networks without skip connections or normalization layers using Deep Kernel Shapingcited by 51
  9. 2019Towards Characterizing Divergence in Deep Q-Learningcited by 43
  10. 2020Efficient Generalized Spherical CNNscited by 29
  11. 2022An Empirical Study of Implicit Regularization in Deep Offline RLcited by 24
  12. 2022Overcoming the Spectral Bias of Neural Value Approximationcited by 18
  13. 2020Improving Performance in Reinforcement Learning by Breaking Generalization in Neural Networkscited by 17
  14. 2021Functional Regularization for Reinforcement Learning via Learned Fourier Featurescited by 17
  15. 2021Correcting Momentum in Temporal Difference Learningcited by 12
  16. 2022Adaptive Online Value Function Approximation with Waveletscited by 1

Built on by

Cited by year

Citing papers per year: 2024: 1; 2025: 3.

4 papers.

  1. 2024Plastic Learning with Deep Fourier Featurescited by 14
  2. 2025Do We Always Need the Simplicity Bias? Looking for Optimal Inductive Biases in the Wildcited by 13
  3. 2025Balancing Expressivity and Robustness: Constrained Rational Activations for Reinforcement Learningcited by 2
  4. 2025Comparing Behavioural Cloning and Reinforcement Learning for Spacecraft Guidance and Control Networkscited by 0