# Understanding Sinusoidal Neural Networks

arXiv: https://arxiv.org/abs/2212.01833 · HTML: https://mlsanity.com/paper/2212.01833/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 19 references that are in the map.
- First posted on arXiv 4 Dec 2022.

## Builds on (most cited first, 11 of 11)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Implicit Neural Representations with Periodic Activation Functions](https://mlsanity.com/paper/2006.09661/) (2020)
- [Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains](https://mlsanity.com/paper/2006.10739/) (2020)
- [On the Spectral Bias of Neural Networks](https://mlsanity.com/paper/1806.08734/) (2018)
- [A Structured Dictionary Perspective on Implicit Neural Representations](https://mlsanity.com/paper/2112.01917/) (2021)
- [Seeing Implicit Neural Representations as Fourier Series](https://mlsanity.com/paper/2109.00249/) (2021)
- [A Level Set Theory for Neural Implicit Evolution under Explicit Flows](https://mlsanity.com/paper/2204.07159/) (2022)
- [Exploring Differential Geometry in Neural Implicits](https://mlsanity.com/paper/2201.09263/) (2022)
- [Neural Implicit Surface Evolution](https://mlsanity.com/paper/2201.09636/) (2022)
- [Multiresolution Neural Networks for Imaging](https://mlsanity.com/paper/2208.11813/) (2022)
- [Neural Implicit Mapping via Nested Neighborhoods](https://mlsanity.com/paper/2201.09147/) (2022)

## Built on by (most cited first, 4 of 4)

- [Distilling Dataset into Neural Field](https://mlsanity.com/paper/2503.04835/) (2025)
- [Implicit Neural Representation of Tileable Material Textures](https://mlsanity.com/paper/2402.02208/) (2024)
- [EIDOS: Latent-Space Predictive Learning for Time Series Foundation Models](https://mlsanity.com/paper/2602.14024/) (2026)
- [Adaptive Training of INRs via Pruning and Densification](https://mlsanity.com/paper/2510.23943/) (2025)

## Abstract (arXiv)

In this work, we investigate the structure and representation capacity of sinusoidal MLPs - multilayer perceptron networks that use sine as the activation function. These neural networks (known as neural fields) have become fundamental in representing common signals in computer graphics, such as images, signed distance functions, and radiance fields. This success can be primarily attributed to two key properties of sinusoidal MLPs: smoothness and compactness. These functions are smooth because they arise from the composition of affine maps with the sine function. This work provides theoretical results to justify the compactness property of sinusoidal MLPs and provides control mechanisms in the definition and training of these networks. We propose to study a sinusoidal MLP by expanding it as a harmonic sum. First, we observe that its first layer can be seen as a harmonic dictionary, which we call the input sinusoidal neurons. Then, a hidden layer combines this dictionary using an affine map and modulates the outputs using the sine, this results in a special dictionary of sinusoidal neurons. We prove that each of these sinusoidal neurons expands as a harmonic sum producing a large number of new frequencies expressed as integer linear combinations of the input frequencies. Thus, each hidden neuron produces the same frequencies, and the corresponding amplitudes are completely determined by the hidden affine map. We also provide an upper bound and a way of sorting these amplitudes that can control the resulting approximation, allowing us to truncate the corresponding series. Finally, we present applications for training and initialization of sinusoidal MLPs. Additionally, we show that if the input neurons are periodic, then the entire network will be periodic with the same period. We relate these periodic networks with the Fourier series representation.