mlsanityData to 14 Oct 2026

Functional Regularization for Reinforcement Learning via Learned Fourier Features

arXiv 2112.03257 ↗First posted 6 Dec 2021cs.LG cs.AI cs.CVcited by 17

Not on the map. Why?

Abstract (arXiv)

We propose a simple architecture for deep reinforcement learning by embedding inputs into a learned Fourier basis and show that it improves the sample efficiency of both state-based and image-based RL. We perform infinite-width analysis of our architecture using the Neural Tangent Kernel and theoretically show that tuning the initial variance of the Fourier basis is equivalent to functional regularization of the learned deep network. That is, these learned Fourier features allow for adjusting the degree to which networks underfit or overfit different frequencies in the training data, and hence provide a controlled mechanism to improve the stability and performance of RL optimization. Empirically, this allows us to prioritize learning low-frequency functions and speed up learning by reducing networks' susceptibility to noise in the optimization process, such as during Bellman updates. Experiments on standard state-based and image-based RL benchmarks show clear benefits of our architecture over the baselines. Website at https://alexanderli.com/learned-fourier-features

Key facts. Cited by 17 papers in the map (data through 14 Oct 2026). Builds on 30 of its 52 references that are in the map. First posted on arXiv 6 Dec 2021.

Builds on

30 of 52 references.

References · 30

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  4. 2015Continuous control with deep reinforcement learningcited by 3,407
  5. 2013Playing Atari with Deep Reinforcement Learningcited by 3,274
  6. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
  7. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  8. 2020Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domainscited by 2,200
  9. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 2,026
  10. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,745
  11. 2016OpenAI Gymcited by 1,736
  12. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,240
18 more
  1. 2018DeepMind Control Suitecited by 956
  2. 2020CURL: Contrastive Unsupervised Representations for Reinforcement Learningcited by 760
  3. 2018On Lazy Training in Differentiable Programmingcited by 675
  4. 2020Reinforcement Learning with Augmented Datacited by 468
  5. 2019Frequency Principle: Fourier Analysis Sheds Light on Deep Neural Networkscited by 368
  6. 2020Deep learning versus kernel learning: an empirical study of loss landscape geometry and the time evolution of the Neural Tangent Kernelcited by 187
  7. 2018Deep Reinforcement Learning and the Deadly Triadcited by 174
  8. 2019Neural Tangents: Fast and Easy Infinite Neural Networks in Pythoncited by 149
  9. 2020Frequency Bias in Neural Networks for Input of Non-Uniform Densitycited by 140
  10. 2019The Convergence Rate of Neural Networks for Learned Functions of Different Frequenciescited by 135
  11. 2020Implicit Under-Parameterization Inhibits Data-Efficient Deep Reinforcement Learningcited by 134
  12. 2019What Can ResNet Learn Efficiently, Going Beyond Kernels?cited by 91
  13. 2019Diagnosing Bottlenecks in Deep Q-learning Algorithmscited by 73
  14. 2021Spectral Normalisation for Deep Reinforcement Learning: an Optimisation Perspectivecited by 65
  15. 2019Towards Characterizing Divergence in Deep Q-Learningcited by 43
  16. 2020D2RL: Deep Dense Architectures in Reinforcement Learningcited by 42
  17. 2020GRAC: Self-Guided and Self-Regularized Actor-Criticcited by 21
  18. 2018Discovering Transforms: A Tutorial on Circulant Matrices, Circular Convolution, and the Discrete Fourier Transformcited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 5; 2023: 2; 2024: 2; 2025: 6; 2026: 2.

17 papers.

Sort:

By area:

  1. 2022AnyMorph: Learning Transferable Polices By Inferring Agent Morphologycited by 27
  2. 2022Overcoming the Spectral Bias of Neural Value Approximationcited by 19
  3. 2024Plastic Learning with Deep Fourier Featurescited by 14
  4. 2023Learning Energy-Based Prior Model with Diffusion-Amortized MCMCcited by 13
  5. 2025Do We Always Need the Simplicity Bias? Looking for Optimal Inductive Biases in the Wildcited by 13
  6. 2023Elephant Neural Networks: Born to Be a Continual Learnercited by 10
  7. 2022How Does Return Distribution in Distributional Reinforcement Learning Help Optimization?cited by 4
  8. 2024Frequency and Generalisation of Periodic Activation Functions in Reinforcement Learningcited by 4
  9. 2025Bridging the Performance Gap Between Target-Free and Target-Based Reinforcement Learningcited by 4
  10. 2022PIP: Positional-encoding Image Priorcited by 3
  11. 2025Balancing Expressivity and Robustness: Constrained Rational Activations for Reinforcement Learningcited by 2
  12. 2025Use the Online Network If You Can: Towards Fast and Stable Reinforcement Learningcited by 2
5 more
  1. 2022ATLAS: Universal Function Approximator for Memory Retentioncited by 0
  2. 2025Maximum Total Correlation Reinforcement Learningcited by 0
  3. 2025Efficient Reinforcement Learning by Reducing Forgetting with Elephant Activation Functionscited by 0
  4. 2026Closing the gap on tabular data with Fourier and Implicit Categorical Featurescited by 0
  5. 2026Learning Holographic Reduced Representations with Clifford Variational Autoencoderscited by 0