# Periodic Extrapolative Generalisation in Neural Networks

arXiv: https://arxiv.org/abs/2209.10280 · HTML: https://mlsanity.com/paper/2209.10280/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 21 references that are in the map.
- First posted on arXiv 21 Sep 2022.

## Builds on (most cited first, 10 of 10)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Sharpness-Aware Minimization for Efficiently Improving Generalization](https://mlsanity.com/paper/2010.01412/) (2020)
- [A Tutorial on Bayesian Optimization](https://mlsanity.com/paper/1807.02811/) (2018)
- [How Neural Networks Extrapolate: From Feedforward to Graph Neural Networks](https://mlsanity.com/paper/2009.11848/) (2020)
- [Neural Networks Fail to Learn Periodic Functions and How to Fix It](https://mlsanity.com/paper/2006.08195/) (2020)
- [Structure Discovery in Nonparametric Regression through Compositional Kernel Search](https://mlsanity.com/paper/1302.4922/) (2013)
- [Deep Symbolic Regression for Recurrent Sequences](https://mlsanity.com/paper/2201.04600/) (2022)
- [Fourier Neural Networks: A Comparative Study](https://mlsanity.com/paper/1902.03011/) (2019)
- [Conditional Generation of Periodic Signals with Fourier-Based Decoder](https://mlsanity.com/paper/2110.12365/) (2021)
- [FACT: Learning Governing Abstractions Behind Integer Sequences](https://mlsanity.com/paper/2209.09543/) (2022)

## Built on by (most cited first, 3 of 3)

- [FAN: Fourier Analysis Networks](https://mlsanity.com/paper/2410.02675/) (2024)
- [Neural Combinatorial Logic Circuit Synthesis from Input-Output Examples](https://mlsanity.com/paper/2210.16606/) (2022)
- [Function Extrapolation with Neural Networks and Its Application for Manifolds](https://mlsanity.com/paper/2405.10563/) (2024)

## Abstract (arXiv)

The learning of the simplest possible computational pattern -- periodicity -- is an open problem in the research of strong generalisation in neural networks. We formalise the problem of extrapolative generalisation for periodic signals and systematically investigate the generalisation abilities of classical, population-based, and recently proposed periodic architectures on a set of benchmarking tasks. We find that periodic and "snake" activation functions consistently fail at periodic extrapolation, regardless of the trainability of their periodicity parameters. Further, our results show that traditional sequential models still outperform the novel architectures designed specifically for extrapolation, and that these are in turn trumped by population-based training. We make our benchmarking and evaluation toolkit, PerKit, available and easily accessible to facilitate future work in the area.