mlsanityData to 13 Oct 2026

Fast Training of Sinusoidal Neural Fields via Scaling Initialization

arXiv 2410.04779 ↗First posted 7 Oct 2024cs.LG cs.AIcited by 11

Not on the map. Why?

Abstract (arXiv)

Neural fields are an emerging paradigm that represent data as continuous functions parameterized by neural networks. Despite many advantages, neural fields often have a high training cost, which prevents a broader adoption. In this paper, we focus on a popular family of neural fields, called sinusoidal neural fields (SNFs), and study how it should be initialized to maximize the training speed. We find that the standard initialization scheme for SNFs -- designed based on the signal propagation principle -- is suboptimal. In particular, we show that by simply multiplying each weight (except for the last layer) by a constant, we can accelerate SNF training by 10×. This method, coined textitweight scaling, consistently provides a significant speedup over various data domains, allowing the SNFs to train faster than more recently proposed architectures. To understand why the weight scaling works well, we conduct extensive theoretical and empirical analyses which reveal that the weight scaling not only resolves the spectral bias quite effectively but also enjoys a well-conditioned optimization trajectory. The code is available hrefhttps://github.com/effl-lab/Fast-Neural-Fieldshere.

Key facts. Cited by 11 papers in the map (data through 13 Oct 2026). Builds on 61 of its 81 references that are in the map. First posted on arXiv 7 Oct 2024.

Builds on

61 of 81 references.

References · 61

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,093
  2. 2022Instant Neural Graphics Primitives with a Multiresolution Hash Encodingcited by 3,712
  3. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,601
  4. 2022DreamFusion: Text-to-3D using 2D Diffusioncited by 3,019
  5. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,610
  6. 2020Implicit Neural Representations with Periodic Activation Functionscited by 2,225
  7. 2020Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domainscited by 2,183
  8. 2018On the Spectral Bias of Neural Networkscited by 1,329
  9. 2019Scene Representation Networks: Continuous 3D-Structure-Aware Neural Scene Representationscited by 864
  10. 2020pi-GAN: Periodic Implicit Generative Adversarial Networks for 3D-Aware Image Synthesiscited by 708
  11. 2018On Lazy Training in Differentiable Programmingcited by 675
  12. 2021Neural Fields in Visual Computing and Beyondcited by 539
38 more
  1. 2019Frequency Principle: Fourier Analysis Sheds Light on Deep Neural Networkscited by 363
  2. 2020Kernel and Rich Regimes in Overparametrized Modelscited by 305
  3. 2019PyHessian: Neural Networks Through the Lens of the Hessiancited by 291
  4. 2019An Investigation into Neural Net Optimization via Hessian Eigenvalue Densitycited by 289
  5. 2020On the eigenvector bias of Fourier feature networks: From regression to solving multi-scale PDEs with physics-informed neural networkscited by 283
  6. 2020Learned Initializations for Optimizing Coordinate-Based Neural Representationscited by 241
  7. 2023WIRE: Wavelet Implicit Neural Representationscited by 220
  8. 2020MetaSDF: Meta-learning Signed Distance Functionscited by 214
  9. 2021Neural Descriptor Fields: SE(3)-Equivariant Object Representations for Manipulationcited by 206
  10. 2019Fixup Initialization: Residual Learning Without Normalizationcited by 195
  11. 2018Dynamical Isometry and a Mean Field Theory of CNNs: How to Train 10,000-Layer Vanilla Convolutional Neural Networkscited by 176
  12. 2019Towards Understanding the Spectral Bias of Deep Learningcited by 174
  13. 2019Neural Tangents: Fast and Easy Infinite Neural Networks in Pythoncited by 149
  14. 2021BACON: Band-limited Coordinate Networks for Multiscale Scene Representationcited by 142
  15. 2020Frequency Bias in Neural Networks for Input of Non-Uniform Densitycited by 140
  16. 2019The Convergence Rate of Neural Networks for Learned Functions of Different Frequenciescited by 135
  17. 2021Beyond Periodicity: Towards a Unifying Framework for Activations in Coordinate-MLPscited by 135
  18. 2024Challenges in Training PINNs: A Loss Landscape Perspectivecited by 111
  19. 2021A Structured Dictionary Perspective on Implicit Neural Representationscited by 100
  20. 2022On the Implicit Bias in Deep-Learning Algorithmscited by 84
  21. 2022COIN++: Neural Compression Across Modalitiescited by 82
  22. 2023Operator Learning with Neural Fields: Tackling PDEs on General Geometriescited by 81
  23. 2022Transformers as Meta-Learners for Implicit Neural Representationscited by 79
  24. 2023FINER: Flexible spectral-bias tuning in Implicit NEural Representation by Variable-periodic Activation Functionscited by 77
  25. 2021On the Implicit Bias of Initialization Shape: Beyond Infinitesimal Mirror Descentcited by 69
  26. 2023Geographic Location Encoding with Spherical Harmonics and Sinusoidal Representation Networkscited by 61
  27. 2021What can linearized neural networks actually say about generalization?cited by 49
  28. 2021Meta-Learning Sparse Implicit Neural Representationscited by 48
  29. 2024Get rich quick: exact solutions reveal how unbalanced initializations promote rapid feature learningcited by 41
  30. 2021On Measuring and Controlling the Spectral Bias of the Deep Image Priorcited by 36
  31. 2022Generalised Implicit Neural Representationscited by 31
  32. 2020Statistical Guarantees for Regularized Neural Networkscited by 25
  33. 2023Compression with Bayesian Implicit Neural Representationscited by 25
  34. 2024A Sampling Theory Perspective on Activations for Implicit Neural Representationscited by 24
  35. 2023Modality-Agnostic Variational Compression of Implicit Neural Representationscited by 22
  36. 2024Efficient Learning With Sine-Activated Low-rank Matricescited by 22
  37. 2021Periodic Activation Functions Induce Stationaritycited by 21
  38. 2022The Spectral Bias of Polynomial Neural Networkscited by 19

50 most cited of 61 All 61 (JSON)

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2025: 8; 2026: 3.

11 papers.

  1. 2025Stochastic Preconditioning for Neural Field Optimizationcited by 8
  2. 2025MedFuncta: A Unified Framework for Learning Efficient Medical Neural Fieldscited by 7
  3. 2025Optimizing Rank for High-Fidelity Implicit Neural Representationscited by 7
  4. 2025A Physics-Informed Meta-Learning Framework for the Continuous Solution of Parametric PDEs on Arbitrary Geometriescited by 5
  5. 2025Spectral Bottleneck in Sinusoidal Representation Networks: Noise is All You Needcited by 2
  6. 2025Improving Accuracy and Efficiency of Implicit Neural Representations: Making SIREN a WINNERcited by 1
  7. 2025On the Internal Representations of Graph Metanetworkscited by 0
  8. 2025Adaptive Training of INRs via Pruning and Densificationcited by 0
  9. 2026Over-Alignment vs Over-Fitting: The Role of Feature Learning Strength in Generalizationcited by 0
  10. 2026ELMZip: Onboard Satellite Image Compression via Extreme Learning Machines for Efficient Downlinkcited by 0
  11. 2026Stable initialization without the CLTcited by 0