mlsanityData to 14 Oct 2026

Spectral Bottleneck in Sinusoidal Representation Networks: Noise is All You Need

arXiv 2509.09719 ↗First posted 9 Sep 2025eess.AS cs.CV cs.LGcited by 2

Not on the map. Why?

Abstract (arXiv)

This work identifies and attempts to address a fundamental limitation of implicit neural representations with sinusoidal activation. The fitting error of SIRENs is highly sensitive to the target frequency content and to the choice of initialization. In extreme cases, this sensitivity leads to a spectral bottleneck that can result in a zero-valued output. This phenomenon is characterized by analyzing the evolution of activation spectra and the empirical neural tangent kernel (NTK) during the training process. An unfavorable distribution of energy across frequency modes was noted to give rise to this failure mode. Furthermore, the effect of Gaussian perturbations applied to the baseline uniformly initialized weights is examined, showing how these perturbations influence activation spectra and the NTK eigenbasis of SIREN. Overall, initialization emerges as a central factor governing the evolution of SIRENs, indicating the need for adaptive, target-aware strategies as the target length increases and fine-scale detail becomes essential. The proposed weight initialization scheme (WINNER) represents a simple ad hoc step in this direction and demonstrates that fitting accuracy can be significantly improved by modifying the spectral profile of network activations through a target-aware initialization. The approach achieves state-of-the-art performance on audio fitting tasks and yields notable improvements in image fitting tasks.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 42 of its 64 references that are in the map. First posted on arXiv 9 Sep 2025.

Builds on

42 of 64 references.

References · 42

  1. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,177
  2. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
  3. 2020Implicit Neural Representations with Periodic Activation Functionscited by 2,238
  4. 2020Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domainscited by 2,200
  5. 2013Describing Textures in the Wildcited by 2,125
  6. 2018On the Spectral Bias of Neural Networkscited by 1,341
  7. 2017Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictionscited by 804
  8. 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 738
  9. 2016CNN Architectures for Large-Scale Audio Classificationcited by 704
  10. 2019Differentiable Volumetric Rendering: Learning Implicit 3D Representations without 3D Supervisioncited by 700
  11. 2020When and why PINNs fail to train: A neural tangent kernel perspectivecited by 634
  12. 2019SAL: Sign Agnostic Learning of Shapes from Raw Datacited by 390
30 more
  1. 2019Frequency Principle: Fourier Analysis Sheds Light on Deep Neural Networkscited by 368
  2. 2019Noise2Self: Blind Denoising by Self-Supervisioncited by 276
  3. 2018Adversarial Audio Synthesiscited by 225
  4. 2023WIRE: Wavelet Implicit Neural Representationscited by 220
  5. 2019Texture Fields: Learning Texture Representations in Function Spacecited by 210
  6. 2019On the Inductive Bias of Neural Tangent Kernelscited by 177
  7. 2019Towards Understanding the Spectral Bias of Deep Learningcited by 174
  8. 2020Frequency Bias in Neural Networks for Input of Non-Uniform Densitycited by 140
  9. 2021Modulated Periodic Activations for Generalizable Local Functional Representationscited by 140
  10. 2019The Convergence Rate of Neural Networks for Learned Functions of Different Frequenciescited by 135
  11. 2021Beyond Periodicity: Towards a Unifying Framework for Activations in Coordinate-MLPscited by 135
  12. 2020Neural Networks Fail to Learn Periodic Functions and How to Fix Itcited by 100
  13. 2021A Structured Dictionary Perspective on Implicit Neural Representationscited by 100
  14. 2022GARF: Gaussian Activated Radiance Fields for High Fidelity Reconstruction and Pose Estimationcited by 86
  15. 2023FINER: Flexible spectral-bias tuning in Implicit NEural Representation by Variable-periodic Activation Functionscited by 78
  16. 2021DiGS : Divergence guided shape implicit neural representation for unoriented point cloudscited by 77
  17. 2021Deep Kronecker neural networks: A general framework for neural networks with adaptive activation functionscited by 49
  18. 2022Generalizable Implicit Neural Representations via Instance Pattern Composerscited by 39
  19. 2024Improved Implicit Neural Representation with Fourier Reparameterized Trainingcited by 36
  20. 2023Disorder-invariant Implicit Neural Representationcited by 27
  21. 2024A Sampling Theory Perspective on Activations for Implicit Neural Representationscited by 24
  22. 2024Tuning the Frequencies: Robust Training for Sinusoidal Neural Networkscited by 19
  23. 2024HOSC: A Periodic Activation Function for Preserving Sharp Features in Implicit Neural Representationscited by 14
  24. 2023Siamese SIREN: Audio Compression with Implicit Neural Representationscited by 13
  25. 2022HyperSound: Generating Implicit Neural Representations of Audio Signals with Hypernetworkscited by 12
  26. 2024FINER++: Building a Family of Variable-periodic Functions for Activating Implicit Neural Representationcited by 11
  27. 2024Fast Training of Sinusoidal Neural Fields via Scaling Initializationcited by 11
  28. 2024ASMR: Activation-sharing Multi-resolution Coordinate Networks For Efficient Inferencecited by 9
  29. 2024Inductive Gradient Adjustment For Spectral Bias In Implicit Neural Representationscited by 8
  30. 2025A Unified Theory of Sinusoidal Activation Families for Implicit Neural Representationscited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 1.

2 papers.

  1. 2025Improving Accuracy and Efficiency of Implicit Neural Representations: Making SIREN a WINNERcited by 1
  2. 2026Hybrid Quantum-Classical PINNs for Scientific Computing: A Multi-GPU Open-Source Frameworkcited by 0