# Spectral Bottleneck in Sinusoidal Representation Networks: Noise is All You Need

arXiv: https://arxiv.org/abs/2509.09719 · HTML: https://mlsanity.com/paper/2509.09719/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 42 of its 64 references that are in the map.
- First posted on arXiv 9 Sep 2025.

## Builds on (most cited first, 30 of 42)

- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Neural Tangent Kernel: Convergence and Generalization in Neural Networks](https://mlsanity.com/paper/1806.07572/) (2018)
- [Implicit Neural Representations with Periodic Activation Functions](https://mlsanity.com/paper/2006.09661/) (2020)
- [Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains](https://mlsanity.com/paper/2006.10739/) (2020)
- [Describing Textures in the Wild](https://mlsanity.com/paper/1311.3618/) (2013)
- [On the Spectral Bias of Neural Networks](https://mlsanity.com/paper/1806.08734/) (2018)
- [Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictions](https://mlsanity.com/paper/1712.05884/) (2017)
- [Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descent](https://mlsanity.com/paper/1902.06720/) (2019)
- [CNN Architectures for Large-Scale Audio Classification](https://mlsanity.com/paper/1609.09430/) (2016)
- [Differentiable Volumetric Rendering: Learning Implicit 3D Representations without 3D Supervision](https://mlsanity.com/paper/1912.07372/) (2019)
- [When and why PINNs fail to train: A neural tangent kernel perspective](https://mlsanity.com/paper/2007.14527/) (2020)
- [SAL: Sign Agnostic Learning of Shapes from Raw Data](https://mlsanity.com/paper/1911.10414/) (2019)
- [Frequency Principle: Fourier Analysis Sheds Light on Deep Neural Networks](https://mlsanity.com/paper/1901.06523/) (2019)
- [Noise2Self: Blind Denoising by Self-Supervision](https://mlsanity.com/paper/1901.11365/) (2019)
- [Adversarial Audio Synthesis](https://mlsanity.com/paper/1802.04208/) (2018)
- [WIRE: Wavelet Implicit Neural Representations](https://mlsanity.com/paper/2301.05187/) (2023)
- [Texture Fields: Learning Texture Representations in Function Space](https://mlsanity.com/paper/1905.07259/) (2019)
- [On the Inductive Bias of Neural Tangent Kernels](https://mlsanity.com/paper/1905.12173/) (2019)
- [Towards Understanding the Spectral Bias of Deep Learning](https://mlsanity.com/paper/1912.01198/) (2019)
- [Frequency Bias in Neural Networks for Input of Non-Uniform Density](https://mlsanity.com/paper/2003.04560/) (2020)
- [Modulated Periodic Activations for Generalizable Local Functional Representations](https://mlsanity.com/paper/2104.03960/) (2021)
- [The Convergence Rate of Neural Networks for Learned Functions of Different Frequencies](https://mlsanity.com/paper/1906.00425/) (2019)
- [Beyond Periodicity: Towards a Unifying Framework for Activations in Coordinate-MLPs](https://mlsanity.com/paper/2111.15135/) (2021)
- [Neural Networks Fail to Learn Periodic Functions and How to Fix It](https://mlsanity.com/paper/2006.08195/) (2020)
- [A Structured Dictionary Perspective on Implicit Neural Representations](https://mlsanity.com/paper/2112.01917/) (2021)
- [GARF: Gaussian Activated Radiance Fields for High Fidelity Reconstruction and Pose Estimation](https://mlsanity.com/paper/2204.05735/) (2022)
- [FINER: Flexible spectral-bias tuning in Implicit NEural Representation by Variable-periodic Activation Functions](https://mlsanity.com/paper/2312.02434/) (2023)
- [DiGS : Divergence guided shape implicit neural representation for unoriented point clouds](https://mlsanity.com/paper/2106.10811/) (2021)
- [Deep Kronecker neural networks: A general framework for neural networks with adaptive activation functions](https://mlsanity.com/paper/2105.09513/) (2021)
- [Generalizable Implicit Neural Representations via Instance Pattern Composers](https://mlsanity.com/paper/2211.13223/) (2022)

## Built on by (most cited first, 2 of 2)

- [Improving Accuracy and Efficiency of Implicit Neural Representations: Making SIREN a WINNER](https://mlsanity.com/paper/2509.12980/) (2025)
- [Hybrid Quantum-Classical PINNs for Scientific Computing: A Multi-GPU Open-Source Framework](https://mlsanity.com/paper/2604.15645/) (2026)

## Abstract (arXiv)

This work identifies and attempts to address a fundamental limitation of implicit neural representations with sinusoidal activation. The fitting error of SIRENs is highly sensitive to the target frequency content and to the choice of initialization. In extreme cases, this sensitivity leads to a spectral bottleneck that can result in a zero-valued output. This phenomenon is characterized by analyzing the evolution of activation spectra and the empirical neural tangent kernel (NTK) during the training process. An unfavorable distribution of energy across frequency modes was noted to give rise to this failure mode. Furthermore, the effect of Gaussian perturbations applied to the baseline uniformly initialized weights is examined, showing how these perturbations influence activation spectra and the NTK eigenbasis of SIREN. Overall, initialization emerges as a central factor governing the evolution of SIRENs, indicating the need for adaptive, target-aware strategies as the target length increases and fine-scale detail becomes essential. The proposed weight initialization scheme (WINNER) represents a simple ad hoc step in this direction and demonstrates that fitting accuracy can be significantly improved by modifying the spectral profile of network activations through a target-aware initialization. The approach achieves state-of-the-art performance on audio fitting tasks and yields notable improvements in image fitting tasks.