mlsanityData to 7 Oct 2026

Neural Networks Fail to Learn Periodic Functions and How to Fix It

arXiv 2006.08195 ↗First posted 15 Jun 2020cs.LG stat.MLcited by 97

Not on the map. Why?

Abstract (arXiv)

Previous literature offers limited clues on how to learn a periodic function using modern neural networks. We start with a study of the extrapolation properties of neural networks; we prove and demonstrate experimentally that the standard activations functions, such as ReLU, tanh, sigmoid, along with their variants, all fail to learn to extrapolate simple periodic functions. We hypothesize that this is due to their lack of a "periodic" inductive bias. As a fix of this problem, we propose a new activation, namely, x + sin2(x), which achieves the desired periodic inductive bias to learn a periodic function while maintaining a favorable optimization property of the ReLU-based activations. Experimentally, we apply the proposed method to temperature and financial data prediction.

Key facts. Cited by 97 papers in the map (data through 7 Oct 2026). First posted on arXiv 15 Jun 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 4; 2022: 13; 2023: 13; 2024: 24; 2025: 29; 2026: 14.

97 papers.

Sort:

By area:

  1. 2023High-Fidelity Audio Compression with Improved RVQGANcited by 423
  2. 2022BigVGAN: A Universal Neural Vocoder with Large-Scale Trainingcited by 245
  3. 2024Stable Audio Opencited by 192
  4. 2024Fast Timing-Conditioned Latent Audio Diffusioncited by 178
  5. 2023Vocos: Closing the gap between time-domain and Fourier-based neural vocoders for high-quality audio synthesiscited by 150
  6. 2023StyleTTS 2: Towards Human-Level Text-to-Speech through Style Diffusion and Adversarial Training with Large Speech Language Modelscited by 148
  7. 2024Long-form music generation with latent diffusioncited by 88
  8. 2022Sparse Mixture-of-Experts are Domain Generalizable Learnerscited by 85
  9. 2025M-Prometheus: A Suite of Open Multilingual LLM Judgescited by 37
  10. 2021Fast Neural Representations for Direct Volume Renderingcited by 32
  11. 2024The Codecfake Dataset and Countermeasures for the Universally Detection of Deepfake Audiocited by 31
  12. 2024EzAudio: Enhancing Text-to-Audio Generation with Efficient Diffusion Transformercited by 30
38 more
  1. 2022Deep Learning and Symbolic Regression for Discovering Parametric Equationscited by 22
  2. 2025FocalCodec: Low-Bitrate Speech Coding via Focal Modulation Networkscited by 18
  3. 2022AERO: Audio Super Resolution in the Spectral Domaincited by 17
  4. 2023Confronting Ambiguity in 6D Object Pose Estimation via Score-Based Diffusion on SE(3)cited by 17
  5. 2025NanoCodec: Towards High-Quality Ultra Fast Speech LLM Inferencecited by 14
  6. 2022Effects of Data Geometry in Early Deep Learningcited by 12
  7. 2023Squared Neural Families: A New Class of Tractable Density Modelscited by 12
  8. 2023FSD: An Initial Chinese Dataset for Fake Song Detectioncited by 12
  9. 2024FAN: Fourier Analysis Networkscited by 12
  10. 2023HiFTNet: A Fast High-Quality Neural Vocoder with Harmonic-plus-Noise Filter and Inverse Short Time Fourier Transformcited by 10
  11. 2024On the Ability of Deep Networks to Learn Symmetries from Data: A Neural Kernel Theorycited by 10
  12. 2022Physical Activation Functions (PAFs): An Approach for More Efficient Induction of Physics into Physics-Informed Neural Networks (PINNs)cited by 9
  13. 2024RFWave: Multi-band Rectified Flow for Audio Waveform Reconstructioncited by 9
  14. 2024Can Transformers Do Enumerative Geometry?cited by 8
  15. 2024Koopman-informed recurrent neural networkscited by 8
  16. 2025When Do Neural Networks Learn World Models?cited by 8
  17. 2025Towards Interactive Intelligence for Digital Humanscited by 8
  18. 2022An unsupervised latent/output physics-informed convolutional-LSTM network for solving partial differential equations using peridynamic differential operatorcited by 7
  19. 2025Speech Enhancement Using Continuous Embeddings of Neural Audio Codeccited by 7
  20. 2021Conditional Generation of Periodic Signals with Fourier-Based Decodercited by 6
  21. 2023BigVSAN: Enhancing GAN-based Neural Vocoders with Slicing Adversarial Networkcited by 6
  22. 2024Exact, Fast and Expressive Poisson Point Processes via Squared Neural Familiescited by 6
  23. 2024vec2wav 2.0: Advancing Voice Conversion via Discrete Token Vocoderscited by 6
  24. 2024Variable Bitrate Residual Vector Quantization for Audio Codingcited by 6
  25. 2023Recent Advancements in Deep Learning Applications and Methods for Autonomous Navigation: A Comprehensive Reviewcited by 5
  26. 2024Codecfake: An Initial Dataset for Detecting LLM-based Deepfake Audiocited by 5
  27. 2022Neural Solvers for Fast and Accurate Numerical Optimal Controlcited by 4
  28. 2022Learning Continuous Implicit Representation for Near-Periodic Patternscited by 4
  29. 2024Towards Audio Codec-based Speech Separationcited by 4
  30. 2024KAN we improve on HEP classification tasks? Kolmogorov-Arnold Networks applied to an LHC physics examplecited by 4
  31. 2025Fourier Multi-Component and Multi-Layer Neural Networks: Unlocking High-Frequency Potentialcited by 4
  32. 2025Demystifying Spectral Feature Learning for Instrumental Variable Regressioncited by 4
  33. 2025PDEfuncta: Spectrally-Aware Neural Representation for PDE Solution Modelingcited by 4
  34. 2025UniFlow: Unifying Speech Front-End Tasks via Continuous Generative Modelingcited by 4
  35. 2022Stochastic Neural Networks with Infinite Width are Deterministiccited by 3
  36. 2022Periodic Extrapolative Generalisation in Neural Networkscited by 3
  37. 2024SpecDiff-GAN: A Spectrally-Shaped Noise Diffusion GAN for Speech and Music Synthesiscited by 3
  38. 2024FlowMAC: Conditional Flow Matching for Audio Coding at Low Bit Ratescited by 3

50 most cited of 97 All 97 (JSON)