# Neural Networks Fail to Learn Periodic Functions and How to Fix It

arXiv: https://arxiv.org/abs/2006.08195 · HTML: https://mlsanity.com/paper/2006.08195/

## Key facts

- Cited by 100 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 15 Jun 2020.

## Built on by (most cited first, 30 of 100)

- [High-Fidelity Audio Compression with Improved RVQGAN](https://mlsanity.com/paper/2306.06546/) (2023)
- [BigVGAN: A Universal Neural Vocoder with Large-Scale Training](https://mlsanity.com/paper/2206.04658/) (2022)
- [Stable Audio Open](https://mlsanity.com/paper/2407.14358/) (2024)
- [Fast Timing-Conditioned Latent Audio Diffusion](https://mlsanity.com/paper/2402.04825/) (2024)
- [Vocos: Closing the gap between time-domain and Fourier-based neural vocoders for high-quality audio synthesis](https://mlsanity.com/paper/2306.00814/) (2023)
- [StyleTTS 2: Towards Human-Level Text-to-Speech through Style Diffusion and Adversarial Training with Large Speech Language Models](https://mlsanity.com/paper/2306.07691/) (2023)
- [Long-form music generation with latent diffusion](https://mlsanity.com/paper/2404.10301/) (2024)
- [Sparse Mixture-of-Experts are Domain Generalizable Learners](https://mlsanity.com/paper/2206.04046/) (2022)
- [M-Prometheus: A Suite of Open Multilingual LLM Judges](https://mlsanity.com/paper/2504.04953/) (2025)
- [Fast Neural Representations for Direct Volume Rendering](https://mlsanity.com/paper/2112.01579/) (2021)
- [The Codecfake Dataset and Countermeasures for the Universally Detection of Deepfake Audio](https://mlsanity.com/paper/2405.04880/) (2024)
- [EzAudio: Enhancing Text-to-Audio Generation with Efficient Diffusion Transformer](https://mlsanity.com/paper/2409.10819/) (2024)
- [Deep Learning and Symbolic Regression for Discovering Parametric Equations](https://mlsanity.com/paper/2207.00529/) (2022)
- [FocalCodec: Low-Bitrate Speech Coding via Focal Modulation Networks](https://mlsanity.com/paper/2502.04465/) (2025)
- [AERO: Audio Super Resolution in the Spectral Domain](https://mlsanity.com/paper/2211.12232/) (2022)
- [Confronting Ambiguity in 6D Object Pose Estimation via Score-Based Diffusion on SE(3)](https://mlsanity.com/paper/2305.15873/) (2023)
- [NanoCodec: Towards High-Quality Ultra Fast Speech LLM Inference](https://mlsanity.com/paper/2508.05835/) (2025)
- [FAN: Fourier Analysis Networks](https://mlsanity.com/paper/2410.02675/) (2024)
- [Effects of Data Geometry in Early Deep Learning](https://mlsanity.com/paper/2301.00008/) (2022)
- [Squared Neural Families: A New Class of Tractable Density Models](https://mlsanity.com/paper/2305.13552/) (2023)
- [FSD: An Initial Chinese Dataset for Fake Song Detection](https://mlsanity.com/paper/2309.02232/) (2023)
- [HiFTNet: A Fast High-Quality Neural Vocoder with Harmonic-plus-Noise Filter and Inverse Short Time Fourier Transform](https://mlsanity.com/paper/2309.09493/) (2023)
- [Physical Activation Functions (PAFs): An Approach for More Efficient Induction of Physics into Physics-Informed Neural Networks (PINNs)](https://mlsanity.com/paper/2205.14630/) (2022)
- [On the Ability of Deep Networks to Learn Symmetries from Data: A Neural Kernel Theory](https://mlsanity.com/paper/2412.11521/) (2024)
- [Towards Interactive Intelligence for Digital Humans](https://mlsanity.com/paper/2512.13674/) (2025)
- [RFWave: Multi-band Rectified Flow for Audio Waveform Reconstruction](https://mlsanity.com/paper/2403.05010/) (2024)
- [Can Transformers Do Enumerative Geometry?](https://mlsanity.com/paper/2408.14915/) (2024)
- [Koopman-informed recurrent neural networks](https://mlsanity.com/paper/2410.23467/) (2024)
- [When Do Neural Networks Learn World Models?](https://mlsanity.com/paper/2502.09297/) (2025)
- [An unsupervised latent/output physics-informed convolutional-LSTM network for solving partial differential equations using peridynamic differential operator](https://mlsanity.com/paper/2210.12177/) (2022)

## Abstract (arXiv)

Previous literature offers limited clues on how to learn a periodic function using modern neural networks. We start with a study of the extrapolation properties of neural networks; we prove and demonstrate experimentally that the standard activations functions, such as ReLU, tanh, sigmoid, along with their variants, all fail to learn to extrapolate simple periodic functions. We hypothesize that this is due to their lack of a "periodic" inductive bias. As a fix of this problem, we propose a new activation, namely, x + sin2(x), which achieves the desired periodic inductive bias to learn a periodic function while maintaining a favorable optimization property of the ReLU-based activations. Experimentally, we apply the proposed method to temperature and financial data prediction.