# Tuning the Frequencies: Robust Training for Sinusoidal Neural Networks

arXiv: https://arxiv.org/abs/2407.21121 · HTML: https://mlsanity.com/paper/2407.21121/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 37 references that are in the map.
- First posted on arXiv 30 Jul 2024.

## Builds on (most cited first, 23 of 23)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Implicit Neural Representations with Periodic Activation Functions](https://mlsanity.com/paper/2006.09661/) (2020)
- [Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains](https://mlsanity.com/paper/2006.10739/) (2020)
- [On the Spectral Bias of Neural Networks](https://mlsanity.com/paper/1806.08734/) (2018)
- [pi-GAN: Periodic Implicit Generative Adversarial Networks for 3D-Aware Image Synthesis](https://mlsanity.com/paper/2012.00926/) (2020)
- [Learning Continuous Image Representation with Local Implicit Image Function](https://mlsanity.com/paper/2012.09161/) (2020)
- [Neural Fields in Visual Computing and Beyond](https://mlsanity.com/paper/2111.11426/) (2021)
- [COIN: COmpression with Implicit Neural representations](https://mlsanity.com/paper/2103.03123/) (2021)
- [WIRE: Wavelet Implicit Neural Representations](https://mlsanity.com/paper/2301.05187/) (2023)
- [BACON: Band-limited Coordinate Networks for Multiscale Scene Representation](https://mlsanity.com/paper/2112.04645/) (2021)
- [A Structured Dictionary Perspective on Implicit Neural Representations](https://mlsanity.com/paper/2112.01917/) (2021)
- [FINER: Flexible spectral-bias tuning in Implicit NEural Representation by Variable-periodic Activation Functions](https://mlsanity.com/paper/2312.02434/) (2023)
- [Geometry-Consistent Neural Shape Representation with Implicit Displacement Fields](https://mlsanity.com/paper/2106.05187/) (2021)
- [Seeing Implicit Neural Representations as Fourier Series](https://mlsanity.com/paper/2109.00249/) (2021)
- [A Level Set Theory for Neural Implicit Evolution under Explicit Flows](https://mlsanity.com/paper/2204.07159/) (2022)
- [Neural Fourier Filter Bank](https://mlsanity.com/paper/2212.01735/) (2022)
- [Exploring Differential Geometry in Neural Implicits](https://mlsanity.com/paper/2201.09263/) (2022)
- [A Sampling Theory Perspective on Activations for Implicit Neural Representations](https://mlsanity.com/paper/2402.05427/) (2024)
- [Polynomial Neural Fields for Subband Decomposition and Manipulation](https://mlsanity.com/paper/2302.04862/) (2023)
- [Neural Implicit Surface Evolution](https://mlsanity.com/paper/2201.09636/) (2022)
- [BANF: Band-limited Neural Fields for Levels of Detail Reconstruction](https://mlsanity.com/paper/2404.13024/) (2024)
- [Neural Implicit Morphing of Face Images](https://mlsanity.com/paper/2308.13888/) (2023)
- [Positional Embedding-Aware Activations](https://mlsanity.com/paper/2306.15242/) (2023)

## Built on by (most cited first, 19 of 19)

- [On the expressiveness and spectral bias of KANs](https://mlsanity.com/paper/2410.01803/) (2024)
- [A Unified Theory of Sinusoidal Activation Families for Implicit Neural Representations](https://mlsanity.com/paper/2502.00869/) (2025)
- [SASNet: Spatially-Adaptive Sinusoidal Networks for INRs](https://mlsanity.com/paper/2503.09750/) (2025)
- [FLOWING: Implicit Neural Flows for Structure-Preserving Morphing](https://mlsanity.com/paper/2510.09537/) (2025)
- [Spectral Bottleneck in Sinusoidal Representation Networks: Noise is All You Need](https://mlsanity.com/paper/2509.09719/) (2025)
- [FM-SIREN & FM-FINER: Implicit Neural Representation Using Nyquist-based Orthogonality](https://mlsanity.com/paper/2509.23438/) (2025)
- [COSMO-INR: Complex Sinusoidal Modulation for Implicit Neural Representations](https://mlsanity.com/paper/2505.11640/) (2025)
- [Improving Accuracy and Efficiency of Implicit Neural Representations: Making SIREN a WINNER](https://mlsanity.com/paper/2509.12980/) (2025)
- [On the accuracy of implicit neural representations for cardiovascular anatomies and hemodynamic fields](https://mlsanity.com/paper/2510.20970/) (2025)
- [A new initialisation to Control Gradients in Sinusoidal Neural network](https://mlsanity.com/paper/2512.06427/) (2025)
- [Adaptive Training of INRs via Pruning and Densification](https://mlsanity.com/paper/2510.23943/) (2025)
- [Singularity Formation: Synergy in Theoretical, Numerical and Machine Learning Approaches](https://mlsanity.com/paper/2604.16842/) (2026)
- [SAND: Spatially Adaptive Network Depth for Fast Sampling of Neural Implicit Surfaces](https://mlsanity.com/paper/2604.25936/) (2026)
- [Spectral Energy Centroid: a Metric for Improving Performance and Analyzing Spectral Bias in Implicit Neural Representations](https://mlsanity.com/paper/2605.12709/) (2026)
- [ImplicitTerrainV2: Wavelet-Guided Spatially Adaptive Neural Terrain Representation](https://mlsanity.com/paper/2605.22556/) (2026)
- [Jacobi-Anger Method for Deterministic Initialization in Implicit Neural Representation](https://mlsanity.com/paper/2606.06671/) (2026)
- [Recurrent Sinusoidal INRs for Efficient High-Fidelity Representation](https://mlsanity.com/paper/2607.21485/) (2026)
- [M-plicits: Neural Implicit Surfaces via Nested Multiscale Residuals](https://mlsanity.com/paper/2609.28684/) (2026)
- [Stable initialization without the CLT](https://mlsanity.com/paper/2609.30633/) (2026)

## Abstract (arXiv)

Sinusoidal neural networks have been shown effective as implicit neural representations (INRs) of low-dimensional signals, due to their smoothness and high representation capacity. However, initializing and training them remain empirical tasks which lack on deeper understanding to guide the learning process. To fill this gap, our work introduces a theoretical framework that explains the capacity property of sinusoidal networks and offers robust control mechanisms for initialization and training. Our analysis is based on a novel amplitude-phase expansion of the sinusoidal multilayer perceptron, showing how its layer compositions produce a large number of new frequencies expressed as integer combinations of the input frequencies. This relationship can be directly used to initialize the input neurons, as a form of spectral sampling, and to bound the network's spectrum while training. Our method, referred to as TUNER (TUNing sinusoidal nEtwoRks), greatly improves the stability and convergence of sinusoidal INR training, leading to detailed reconstructions, while preventing overfitting.