mlsanityData to 7 Oct 2026

From Activation to Initialization: Scaling Insights for Optimizing Neural Fields

arXiv 2403.19205 ↗First posted 28 Mar 2024cs.CV cs.LGcited by 11

Not on the map. Why?

Abstract (arXiv)

In the realm of computer vision, Neural Fields have gained prominence as a contemporary tool harnessing neural networks for signal representation. Despite the remarkable progress in adapting these networks to solve a variety of problems, the field still lacks a comprehensive theoretical framework. This article aims to address this gap by delving into the intricate interplay between initialization and activation, providing a foundational basis for the robust optimization of Neural Fields. Our theoretical insights reveal a deep-seated connection among network initialization, architectural choices, and the optimization process, emphasizing the need for a holistic approach when designing cutting-edge Neural Fields.

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Builds on 27 of its 37 references that are in the map. First posted on arXiv 28 Mar 2024.

Builds on

27 of 37 references.

References · 27

  1. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,590
  2. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  3. 2020Implicit Neural Representations with Periodic Activation Functionscited by 2,214
  4. 2020Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domainscited by 2,170
  5. 2019Scene Representation Networks: Continuous 3D-Structure-Aware Neural Scene Representationscited by 864
  6. 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 784
  7. 2020Implicit Geometric Regularization for Learning Shapescited by 707
  8. 2021KiloNeRF: Speeding up Neural Radiance Fields with Thousands of Tiny MLPscited by 672
  9. 2021BARF: Bundle-Adjusting Neural Radiance Fieldscited by 641
  10. 2018Gradient Descent Finds Global Minima of Deep Neural Networkscited by 572
  11. 2019Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networkscited by 562
  12. 2020Learning Continuous Image Representation with Local Implicit Image Functioncited by 550
15 more
  1. 2018Learning and Generalization in Overparameterized Neural Networks, Going Beyond Two Layerscited by 340
  2. 2023WIRE: Wavelet Implicit Neural Representationscited by 220
  3. 2019Towards moderate overparameterization: global convergence guarantees for training shallow neural networkscited by 176
  4. 2020Adversarial Generation of Continuous Imagescited by 135
  5. 2021Beyond Periodicity: Towards a Unifying Framework for Activations in Coordinate-MLPscited by 134
  6. 2019An Improved Analysis of Training Over-parameterized Deep Neural Networkscited by 120
  7. 20213D Neural Scene Representations for Visuomotor Controlcited by 110
  8. 2019Dynamics of Deep Neural Networks and Neural Tangent Hierarchycited by 96
  9. 2022GARF: Gaussian Activated Radiance Fields for High Fidelity Reconstruction and Pose Estimationcited by 86
  10. 2021On the Proof of Global Convergence of Gradient Descent for Deep ReLU Networks with Linear Widthscited by 44
  11. 2018The loss landscape of overparameterized neural networkscited by 38
  12. 2021CoIL: Coordinate-based Internal Learning for Imaging Inverse Problemscited by 33
  13. 2021Spline Positional Encoding for Learning 3D Implicit Signed Distance Fieldscited by 21
  14. 2023Curvature-Aware Training for Coordinate Networkscited by 8
  15. 2023On the effectiveness of neural priors in modeling dynamical systemscited by 4

Built on by

Cited by year

Citing papers per year: 2024: 2; 2025: 9.

11 papers.

  1. 2024Fast Training of Sinusoidal Neural Fields via Scaling Initializationcited by 11
  2. 2024Inductive Gradient Adjustment For Spectral Bias In Implicit Neural Representationscited by 8
  3. 2025Optimizing Rank for High-Fidelity Implicit Neural Representationscited by 7
  4. 2025Enhancing Transformers Through Conditioned Embedded Tokenscited by 3
  5. 2025Data Denoising and Derivative Estimation for Data-Driven Modeling of Nonlinear Dynamical Systemscited by 1
  6. 2025From Tables to Signals: Revealing Spectral Adaptivity in TabPFNcited by 1
  7. 2025Strategies for training point distributions in physics-informed neural networkscited by 0
  8. 2025AIM: Adaptive Intra-Network Modulation for Balanced Multimodal Learningcited by 0
  9. 2025Adaptive Training of INRs via Pruning and Densificationcited by 0
  10. 2025Dynamical Implicit Neural Representationscited by 0
  11. 2025Towards High-Fidelity Gaussian Splatting with Queried-Convolution Neural Networkscited by 0