# Functional Regularization for Reinforcement Learning via Learned Fourier Features

arXiv: https://arxiv.org/abs/2112.03257 · HTML: https://mlsanity.com/paper/2112.03257/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 52 references that are in the map.
- First posted on arXiv 6 Dec 2021.

## Builds on (most cited first, 30 of 30)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [Playing Atari with Deep Reinforcement Learning](https://mlsanity.com/paper/1312.5602/) (2013)
- [Neural Tangent Kernel: Convergence and Generalization in Neural Networks](https://mlsanity.com/paper/1806.07572/) (2018)
- [Addressing Function Approximation Error in Actor-Critic Methods](https://mlsanity.com/paper/1802.09477/) (2018)
- [Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains](https://mlsanity.com/paper/2006.10739/) (2020)
- [High-Dimensional Continuous Control Using Generalized Advantage Estimation](https://mlsanity.com/paper/1506.02438/) (2015)
- [Deep Reinforcement Learning with Double Q-learning](https://mlsanity.com/paper/1509.06461/) (2015)
- [OpenAI Gym](https://mlsanity.com/paper/1606.01540/) (2016)
- [Soft Actor-Critic Algorithms and Applications](https://mlsanity.com/paper/1812.05905/) (2018)
- [DeepMind Control Suite](https://mlsanity.com/paper/1801.00690/) (2018)
- [CURL: Contrastive Unsupervised Representations for Reinforcement Learning](https://mlsanity.com/paper/2004.04136/) (2020)
- [On Lazy Training in Differentiable Programming](https://mlsanity.com/paper/1812.07956/) (2018)
- [Reinforcement Learning with Augmented Data](https://mlsanity.com/paper/2004.14990/) (2020)
- [Frequency Principle: Fourier Analysis Sheds Light on Deep Neural Networks](https://mlsanity.com/paper/1901.06523/) (2019)
- [Deep learning versus kernel learning: an empirical study of loss landscape geometry and the time evolution of the Neural Tangent Kernel](https://mlsanity.com/paper/2010.15110/) (2020)
- [Deep Reinforcement Learning and the Deadly Triad](https://mlsanity.com/paper/1812.02648/) (2018)
- [Neural Tangents: Fast and Easy Infinite Neural Networks in Python](https://mlsanity.com/paper/1912.02803/) (2019)
- [Frequency Bias in Neural Networks for Input of Non-Uniform Density](https://mlsanity.com/paper/2003.04560/) (2020)
- [The Convergence Rate of Neural Networks for Learned Functions of Different Frequencies](https://mlsanity.com/paper/1906.00425/) (2019)
- [Implicit Under-Parameterization Inhibits Data-Efficient Deep Reinforcement Learning](https://mlsanity.com/paper/2010.14498/) (2020)
- [What Can ResNet Learn Efficiently, Going Beyond Kernels?](https://mlsanity.com/paper/1905.10337/) (2019)
- [Diagnosing Bottlenecks in Deep Q-learning Algorithms](https://mlsanity.com/paper/1902.10250/) (2019)
- [Spectral Normalisation for Deep Reinforcement Learning: an Optimisation Perspective](https://mlsanity.com/paper/2105.05246/) (2021)
- [Towards Characterizing Divergence in Deep Q-Learning](https://mlsanity.com/paper/1903.08894/) (2019)
- [D2RL: Deep Dense Architectures in Reinforcement Learning](https://mlsanity.com/paper/2010.09163/) (2020)
- [GRAC: Self-Guided and Self-Regularized Actor-Critic](https://mlsanity.com/paper/2009.08973/) (2020)
- [Discovering Transforms: A Tutorial on Circulant Matrices, Circular Convolution, and the Discrete Fourier Transform](https://mlsanity.com/paper/1805.05533/) (2018)

## Built on by (most cited first, 17 of 17)

- [AnyMorph: Learning Transferable Polices By Inferring Agent Morphology](https://mlsanity.com/paper/2206.12279/) (2022)
- [Overcoming the Spectral Bias of Neural Value Approximation](https://mlsanity.com/paper/2206.04672/) (2022)
- [Plastic Learning with Deep Fourier Features](https://mlsanity.com/paper/2410.20634/) (2024)
- [Learning Energy-Based Prior Model with Diffusion-Amortized MCMC](https://mlsanity.com/paper/2310.03218/) (2023)
- [Do We Always Need the Simplicity Bias? Looking for Optimal Inductive Biases in the Wild](https://mlsanity.com/paper/2503.10065/) (2025)
- [Elephant Neural Networks: Born to Be a Continual Learner](https://mlsanity.com/paper/2310.01365/) (2023)
- [How Does Return Distribution in Distributional Reinforcement Learning Help Optimization?](https://mlsanity.com/paper/2209.14513/) (2022)
- [Frequency and Generalisation of Periodic Activation Functions in Reinforcement Learning](https://mlsanity.com/paper/2407.06756/) (2024)
- [Bridging the Performance Gap Between Target-Free and Target-Based Reinforcement Learning](https://mlsanity.com/paper/2506.04398/) (2025)
- [PIP: Positional-encoding Image Prior](https://mlsanity.com/paper/2211.14298/) (2022)
- [Balancing Expressivity and Robustness: Constrained Rational Activations for Reinforcement Learning](https://mlsanity.com/paper/2507.14736/) (2025)
- [Use the Online Network If You Can: Towards Fast and Stable Reinforcement Learning](https://mlsanity.com/paper/2510.02590/) (2025)
- [ATLAS: Universal Function Approximator for Memory Retention](https://mlsanity.com/paper/2208.05388/) (2022)
- [Maximum Total Correlation Reinforcement Learning](https://mlsanity.com/paper/2505.16734/) (2025)
- [Efficient Reinforcement Learning by Reducing Forgetting with Elephant Activation Functions](https://mlsanity.com/paper/2509.19159/) (2025)
- [Closing the gap on tabular data with Fourier and Implicit Categorical Features](https://mlsanity.com/paper/2602.23182/) (2026)
- [Learning Holographic Reduced Representations with Clifford Variational Autoencoders](https://mlsanity.com/paper/2609.28409/) (2026)

## Abstract (arXiv)

We propose a simple architecture for deep reinforcement learning by embedding inputs into a learned Fourier basis and show that it improves the sample efficiency of both state-based and image-based RL. We perform infinite-width analysis of our architecture using the Neural Tangent Kernel and theoretically show that tuning the initial variance of the Fourier basis is equivalent to functional regularization of the learned deep network. That is, these learned Fourier features allow for adjusting the degree to which networks underfit or overfit different frequencies in the training data, and hence provide a controlled mechanism to improve the stability and performance of RL optimization. Empirically, this allows us to prioritize learning low-frequency functions and speed up learning by reducing networks' susceptibility to noise in the optimization process, such as during Bellman updates. Experiments on standard state-based and image-based RL benchmarks show clear benefits of our architecture over the baselines. Website at https://alexanderli.com/learned-fourier-features